Google Summer of Code 2015

Google Summer of Code 2015

At Twitter, we love Open Source, working with students and Google Summer of Code (GSOC)! What is GSOC? Every year, Google invites students to come up with interesting problems for their favorite open-source projects and work on them over the summer. Participants get support from the community, plus a mentor who makes sure you don't get lost and that you meet your goals. Aside from the satisfaction of solving challenging problems and contributing to the open source community, students get paid and get some sweet swag for their work! In our opinion, this is a great opportunity to get involved with open source, improve your skills and help out the community!

If you're interested in Outreachy (formerly Outreach Program for Women) as an option, please see that wiki: https://github.com/twitter/twitter.github.com/wiki/Outreachy-(Round-10)

Information for Students

These ideas were contributed by our developers and our community, they are only meant to be a starting point. If you wish to submit a proposal based on these ideas, you may wish to contact the developers and find out more about the particular suggestion you're looking at.

Being accepted as a Google Summer of Code student is quite competitive. Accepted students typically have thoroughly researched the technologies of their proposed project and have been in frequent contact with potential mentors. Simply copying and pasting an idea here will not work. On the other hand, creating a completely new idea without first consulting potential mentors is unlikely to work out.

If there is no specific contact given you can ask questions via @TwitterOSS or via the twitter-gsoc mailing list.

Accepted Projects

For 2015, @TwitterOSS accepted X students to work on Y different open source projects:

• TODO

The project details are listed below:

Adding a Proposal

Please follow this template:

• Brief explanation:
• Expected results:
• Knowledge Prerequisite:
• Mentor:

When adding an idea to this section, please try to include the following data.

If you are not a developer but have a good idea for a proposal, get in contact with relevant developers first or @TwitterOSS.

Project Ideas

Finagle

A good starting point is Finagle is the Quickstart: http://twitter.github.io/finagle/guide/Quickstart.html

You could also start digging in the code here: https://github.com/twitter/finagle/

Check out the Finagle mailing list if you have any questions.

Kerberos authentication in Mux

• Brief explanation: Mux is a new RPC session protocol in use at Twitter. We would like to add kerberos authentication.
• Knowledge Prerequisite: Scala, Distributed systems
• Mentor: Marius Eriksen (@marius) and Steve Gury (@stevegury)

Examples and Service Adaptors for Stitch

• Brief explanation: Stitch is a library for RPC service composition that makes it easy to take advantage of batch APIs without muddling up your code with explicit batching logic. We'd like to develop better examples and tools for developers who want to use Stitch in the context of Finagle.
• Knowledge prerequisites: Scala and an interest in learning about Finagle
• Mentors: Travis Brown (@travisbrown)

Alternative Service Representations for Scrooge

• Brief explanation: Scrooge is a Thrift code generator that can create client and server adaptors for Finagle, but the current representation of service interfaces makes it difficult to wrap endpoints in Finagle filters, for example. We're interested in exploring other approaches that would allow Scrooge-generated clients and servers to fit more cleanly into the abstractions provided by Finagle.
• Knowledge prerequisites: Scala and an interest in learning about Thrift and Finagle
• Mentors: Nik Shkrob (@nshkrob)

Aurora

TBD

Mesos

Libprocess Benchmark Suite

• Brief explanation: Implement a benchmark suite for libprocess to identify potential performance improvements and test for performance regressions.
• Knowledge Prerequisite: C++
• Mentor: Ben Mahler (@bmahler) Jie Yu (@jie_yu)
• JIRA ISsue: MESOS-1018

TODO

Summingbird

Summingbird is a library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including Storm and Scalding.

Addition of Akka backend for streaming compute

• Brief explanation: Akka(http://akka.io) is a popular open source distributed actor system. Integrating this into Summingbird would increase the range of potential compute platform for users. Making the system more accessible and suitable for more varied tasks.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Hadoop.
• Mentor: Oscar Boykin (@posco)

Addition of Samza backend for streaming compute

• Brief explanation: Samza(http://samza.incubator.apache.org/) is a new Apache incubator project allowing compute to be placed between two Kafka streams. Integrating this into Summingbird would increase the range of potential compute platform for users. Making the system more accessible and suitable for more varied tasks.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Hadoop, Yarn.
• Mentor: Oscar Boykin (@posco) or Ian O'Connell (@0x138)

Better Spark Support in Summingbird

• Brief explanation: We currently have an alpha version of Spark support for batch computation. This should be completed along with creating a demo application. After that, we should add a realtime layer using spark-streaming.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Spark.
• Mentor: Oscar Boykin (@posco) or Ian O'Connell (@0x138)

Addition of Tez backend for offline batch compute

• Brief explanation: Tez(http://tez.incubator.apache.org) is a new Apache incubator to generalize and expand the map/reduce model of computation. Summingbird should be able to automatically take advantage of map-reduce-reduce plans, and other optimizations that Tez enables. This should perform better than the existing Hadoop-via-cascading-via-scalding backend that is currently available.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Hadoop, Yarn.
• Mentor: Ian O'Connell (@0x138)

Addition of batch key/value store on Mesos or Yarn

• Brief explanation: Something that is sorely missing from the open source release of scalding is a good batch-writable read-only key-value store to use for batch jobs. This could be something like ElephantDB (https://github.com/nathanmarz/elephantdb) or HBase. Having such a project set up with Summingbird would be a huge coup for the open-source community.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Ideally familiar with mesos or yarn, and low latency key-value stores likes HBase or ElephantDB.
• Mentor: Oscar Boykin (@posco) or Ian O'Connell (@0x138)

Scalding

Scalding Twitter's library for programming in scala on Hadoop. It is approachable by new-comers with a fields/Data-frame-like API as well as a type-safe API. There is also a linear algebra API to support working with giant matrices and vectors on Hadoop.

Apace Tez support for Scalding

• Brief explanation: Cascading 3 supports Apache Tez, which may compete in some workloads with spark. If we update Scalding to use Cascading 3, we should be able to run Scalding on Tez. There are lots of little issues here and a few big ones as some concepts from Hadoop are not present in Tez (the distributed cache changes) and some cascading features are not yet supported.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Hadoop. Must be familiar with graphs for modeling flows of computation.
• Mentor: Oscar Boykin @posco

Query Optimization in Scalding

• Right now, Scalding has some optimizations that it can do because it can see the types of the data and functions. Those optimizations are baked into how the graph is produced. This project would instead create an AST in the Typed-API of scalding, and only just before running would we do a global optimization to produce the most optimal cascading plan. This work can leverage existing code created for summingbird to optimize this graphs.
• Knowledge Prerequisite: Need to know scala (or strong knowledge of Java with some functional programming background). Need to be somewhat familiar with Hadoop. Must be familiar with graphs for modeling flows of computation.
• Mentor: Oscar Boykin @posco

Parquet

https://github.com/Parquet/parquet-mr/issues?labels=GSoC-2014&state=open

Use statistics to implement page level filtering in the filter2 API

• Brief explanation: We currently apply filters to entire row groups as well as individual records, but we could apply them to pages as well. This would work similar to how row group filtering currently works.
• Expected results:
  • Statistics based filtering applied to pages in the parquet read path
  • Additional tests for correctness
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Collect more statistics for rowgroups + pages

• Brief explanation: We currently only collect the number of records, number of nulls, min, and max for a chunk of records. We could make use of more statistics when filtering in the read path.
• Expected results:
  • Investigate which statistics would be most useful
  • Add more types of statistics, such as:
    • A bloom filter of the values when a chunk is not dictionary encoded (good for filtering)
    • A HyperLogLog of the values (good for fast count-distinct)
    • A CountMinSketch of the values (good for heavy hitters)
  • Additional tests for correctness
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Add more filters to the filter2 API

• Brief explanation: Parquet can currently filter values by ==, !=, >, >=, <, <= -- we could add some more, for example filter by value in(1,2,3) or notIn(1,2,3)
• Expected results:
  • Add more filter types to the filter2 API
  • Additional tests for correctness
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Take advantage of dictionary encoding in the filter2 API

• Brief explanation: When applying filters to dictionary encoded columns, apply the filter to the dictionary instead of to the individual values.
• Expected results:
  • Use dictionaries when filtering row groups
  • Use dictionaries when filtering individual records
  • Additional tests for correctness
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Profile and improve the assembly time for parquet-thrift

• Brief explanation: When assembling thrift records, parquet uses a TProtocol that boxes every value in an anonymous class. Investigate an implement a more efficient solution.
• Expected results:
  • Faster implementation of assembling thrift records
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Parquet compatibility across tools

• Brief explanation: Develop cross tools compatibility tests for parquet (https://github.com/Parquet/parquet-mr/issues/300)
• Expected results:
  • Compatibility of nested data types across tools - pig, hive, avro, thrift etc.
  • Automated compatibility check between java implementation and impala (across release versions)
• Knowledge Prerequisite: Java, Hadoop, Test frameworks
• Mentor: Alex Levenson (@THISWILLWORK) and/or Julien Le Dem (@J_)

Decouple Parquet from the Hadoop API

• Brief explanation: To allow reading and writing Parquet files independently of the Hadoop API. (https://github.com/Parquet/parquet-mr/issues/305)
• Expected results: read and write Parquet without the Hadoop libraries
• Mentor: Julien Le Dem (@J_)

Study state of the art floating point compression algorithms

(https://github.com/Parquet/parquet-mr/issues/306)

• Brief explanation: Study existing lossless floating point compression papers and implement benchmarks.
• Expected results: Provide reference implementation and benchmark comparison. With integration into the Parquet library
• Mentor: Julien Le Dem (@J_)

Netty

You can learn more about getting involved with the Netty Project here: http://netty.io/community.html

Android testsuite

• Brief explanation:
  • Netty project team is willing to support Android 4.0 Ice Cream Sandwich officially, and we need an automated testsuite to achieve the goal.
• Expected results:
  • During the build process, an Android emulator is automatically started and stopped to run all (or applicable) JUnit tests inside the Android emulator.
  • The result of the JUnit tests inside the emulator affects the build result so that we can run the Android compatibility test in our CI machine.
  • All Android compatibility issues found during the test are fixed.
• Knowledge Prerequisite:
  • Java and Android programming
  • Custom JUnit runners
  • Experience with building a network application atop Netty
• Mentor: Trustin Lee (@trustin)

Pants

For more information about Pants, check these out:

• Contributors Guide: http://pantsbuild.github.io/howto_contribute.html
• Developers Guide: http://pantsbuild.github.io/howto_develop.html
• Task Developers Guide: http://pantsbuild.github.io/dev_tasks.html

Pants Interactive Tutorial

• Brief explanation: Add an interactive tutorial to learn Pants
• Expected results: Create an interactive tutorial that will guide you in learning pants. Use JQueryTerminal to help stage a prompt that guides you through some simple Pants usage cases.
• Knowledge Prerequisite: Python, Java, Javascript
• Mentor: Chris Aniszczyk (@cra)

Eclipse Integration

• Brief explanation: Add Eclipse integration to Pants
• Expected results: Create a classpath container based on integrating with Pants and a launcher.
• Knowledge Prerequisite: Python, Java, Eclipse
• Mentor: Chris Aniszczyk (@cra)

Project

Project URL

Project Idea (e.g., New Feature)

• Brief explanation:
• Expected results:
• Knowledge Prerequisite:
• Mentor:

General Proposal Requirements

Proposals will be submitted via http://www.google-melange.com/gsoc/homepage/google/gsoc2014, therefore plain text is the best way to go. We expect your application to be in the range of 1000 words. Anything less than that will probably not contain enough information for us to determine whether you are the right person for the job. Your proposal should contain at least the following information, but feel free to include anything that you think is relevant:

• Please include your name and twitter handle!
• Title of your proposal
• Abstract of your proposal
• A link to your github id (if you have one)
• Detailed description of your idea including explanation on why is it innovative
• Description of previous work, existing solutions (links to prototypes, bibliography are more than welcome)
• Mention the details of your academic studies, any previous work, internships
• Any relevant skills that will help you to achieve the goal (programming languages, frameworks)?
• Any previous open-source projects (or even previous GSoC) you have contributed to?
• Do you plan to have any other commitments during SoC that may affect you work? Any vacations/holidays planned?
• Contact details

Good luck!