Mango User Guide

Mango provides utility classes for accessing and manipulating the base Java collections. The following table lists the utility class for the given Java type.

The utility classes provide methods for creating new instances and manipulating existing instances. Methods used to create an instance of a given type are defined as follows:

Methods for manipulating collections varying based on type. We refer the reader to the JavaDoc.

A counter is mapping from an Object to a Double where the double value represents a count. A double value is used in case the value represents a normalized count. Mango provides the Counters utility class for constructing new Counters. Currently, Mango provides a HashMapCounter and a ConcurrentHashMapCounter implementation.

In addition to the standard Counters, Mango provides a MultiCounter which maps a tuple of objects to a value. Mango provides the MutliCounters utility class for constructing new MultiCounters. Currently, Mango provides a HashMapMultiCounter and a ConcurrentHashMapMultiCounter implementation.

Both Counters and MultiCounters provide numerous methods to manipulate and query the counts of the objects. These methods include finding the top or bottom N items, filtering by key or value, and determining the minimum and maximum values and their associated objects.

A Multimap maps keys to multiple values. They act as a Map<K, Collection<V> where individual implementations specify the type of collection, e.g. List, Set, etc. Mango currently provides Multimap implementations wrapping the following collection types:

In all cases the backing map used is a HashMap.

Multimaps provide views over the keys and values the same as a Java Map. These views update the underlying Multimap when changed (e.g. items are deleted). Please see the JavaDoc for the full list of methods available on Multimaps.

A table is a two-dimensional structure that associates a value with two keys (i.e. a row and column key). A table maybe sparse, meaning not all cells contain values. Methods on a table that work with rows and columns return Map views that when updated will be reflected in table. Currently, Mango provides a single table implementation, HashBasedTable, which wraps Java’s HashMap.

Mango provides two tree-based datastructures. The first is an IntervalTree which facilitates fast lookup of ranges including overlapping ranges. Interval Trees are a Set-like object which take subclasses of Span as their values. A Span defines a start and end range. Interval tress provide similar methods to those on a NaviagableSet with an additional method overlapping(Span) which provides fast lookup of all Spans in the tree that overlap with the given span.

Mango also provides a basic Trie implementation that facilitates fast prefix lookups in strings. The Trie implements the Map interface where the key is a String and the value can be defined per use. The Trie class provides some useful methods for suggesting the most similar strings given a maximum edit distance and finding all matches of the keys in the Trie in a given String.

Mango provides a basic graph data structure which is currently has one implementation backed by a Table. Mango graphs can be defined as being directed or undirected by defining the EdgeFactory used by the graph. A number of graph algorithms and traversal strategies are implemented including, breadth-fist and depth-first search, Dijkstra’s shortest path, and random walks. Additionally, Mango provides implementations of connected components and Chinese Whispers for clustering. Vertices can be scored by degree, Page Rank, and random walks using one the implementations of VertexScorer. Finally, graphs can be written to json or GraphViz dot format and rendered using GraphViz.

Mango provides a basic set of in-memory Caches to speed up IO intensive processes. The Cache interface is inspired by Guava’s Cache and has methods for getting, putting, and invalidating entries. There are currently two implemenations of Cache: LRUCache which keeps the last N most recently used items and AutoCalculatingLRUCache which extends LRUCache to auto-calculate missing values.

Mango provides a generic interface for key-value stores and provides in-memory and disk-backed versions. A KeyValueStore defines a mapping from keys to values and extends the Java Map interface. Additionally, a NavigableKeyValueStore is defined in Mango that extends the Java NavigableMap interface.

Connections to key-value stores are done through a KeyValueStoreConnection as follows:

The key-value store specification is defined as follows:

Where mem creates an in-memory key-value store and disk a disk-based key-value store. The namespace defines the store name an allows multiple stores to be associated with a single file. The path is only required for disk-based key-value stores and the readOnly parameter is optional denoting if the store is read only (this is false by default).

Streams are created through a StreamingContext. A local streaming context is generated using StreamingContext.local() or through StreamingContext.get(false) whereas a distributed (Spark) contetext is retrieved by StreamingContext.distributed() or StreamingContext.get(false). Streaming contexts provide a variety of ways for creating an MStream, including the following:

Note that Mango also implements a reusable versions of Java’s Stream classes (Stream, IntStream, DoubleStream, and LongStream). The reusable streams use a Supplier to provide the underlying stream. Individual methods, i.e. map then create a new supplier where the return stream applies the given method. Reusable streams can be created outside of MStreams using the Streams utility class.

An accumulator is a variable that can be used for aggregating values in a stream. As with streams, accumulators are created using a streaming context, e.g. StreamingContext.local().counterAccumulator(). Mango provides the following accumulator implementations:

Accumulators can have names associated with them, which will show up in the Spark interface. Additional accumulators can be created by implementing the base MAccumulator interface.

Working with Mango streams is very similar to working with Java streams. The core operations are:

Aggregation can be performed over streams using one of the following methods:

Mango Streams can be converted to distrubted Spark streams by simplying calling toDistributedStream. Similarly, a Java stream can be created using the method javaStream.

Mango aims to make using reflection a little easier. Reflection in Mango starts with the RBase class which defines the base set of functionality for performing reflection on an object, class, method, field, or parameter. This base set of functionality comes in the form of querying the annotations on the reflected item and conditionally performing some action based on the presence of an annotation. The following figure illustrates the hierarchy of classes that defines the Java reflection wrappers.

The main entry point for working with reflection in Mango is the Reflect class. Reflect is a wrapper around an object or class providing easy ways to access the methods, fields, constructors, and annotations on the object/class. An instance is created using on of the static methods as follows:

The Reflect instance respects scope by default. This can be changed by allowing privileged access as follows:

Privileged access will allow the protected and private elments of the object/class to be manipulated.

Once we have an instance of reflect we can query the object/class for its accessible fields as follows:

Similarly, we can query the object/class for its accessible methods as follows:

Similarly, we can query the object/class for its accessible constructors as follows:

The core component of Mango’s Val class and configuration framework is the ability to convert any arbitrary type to another. This conversion is done using the Mango Converter class, which utilizes a number of TypeConverter registered using Java’s Service Loader. A TypeConverter defines the following:

When converting a source object we can us the following methods:

Mango provides many type converters out of the box which cover core Java, java.util, and Mango types. New type converters can be registered using Java’s Service loader. We recommend using the org.kohsuke.metainf-services package to ease this process by only needing to add a @MetaInfServices(value = TypeConverter.class) annotation to your type converter class.

Mango provides the Cast class to easily cast an object. It provides the following basic methods:

Additionally, there are methods for casting the elements of collections, iterables, iterators, and maps: sahi2001

The methods listed above perform casting in lazy manner and do not change the underlying data. These methods are most useful when passing an item as method parameter.

A resource represents a source or destination of/for data. It is similar in idea to Spring’s Resource class. At the core a Resource object facilities opening the data source for reading (InputStream or Reader) and/or writing (OutputStream or Writer) and traversing and manipulating the structure of the data source (e.g. retrieving all children of a folder or the folder for a file, deleting an element, or adding new folders).

The following is a list of the supported resource types and whether they support being read from, written to, or traversed.

Resources are created by either using the constructor of one of the implementations or by using the Resources utlility class. The main way of creating a resource is using Resources.from(String) where the given string defines the resource scheme. Each scheme has an associated ResourceProvider which takes care of parsing the scheme and creating a Resource instance. The scheme is in the following format: PROTOCOL(?OPTIONS):PATH where OPTIONS is optional. For example a FileResource can specified using file:/home/user/file.text. Common options include, the charset and compression technique and are set as follows: file?compression=GZIP,charset=SJIS:/home/user/file.text. Note that options are set using a BeanMap thus the valid options for each resource type are the setters on that type.

The following is a list of Schemes, the generated resource type, and what the path represents for those types accessible via Resources.from(String):

Note that for convenience you can leave off the scheme for file resources, e.g. Resources.from("/home/user/test.csv") will assume the given String is a FileResource.

A common pitfall in Java is not properly closing resources. This can become especially tricky when dealing with concurrency and the new Java stream framework. Mango provides a ResourceMonitor which tracks MonitoredObjects and automatically closes (frees) them when they are no longer referenced. The ResourceMonitor is basically a garbage collector for resources!

The ResourceMonitor class provides convenience methods for monitoring the most common types:

Additionally, it provides a generic monitor method that takes an Object and returns a MonitoredObject wrapping the given object. For generic Objects you can also specify a custom procedure to run when the resource is "closed" by passing in Consumer<T> that will free resources. An example of creating a Monitored object from a custom class is as follows:

The object is wrapped in a MonitoredObject which is tracked by the resource monitor. We have specified a custom on-close operation, which will be called when there are no other references to the wrapped MyClass object. The wrapped object is accessed via the public field object

Delimited Separated Value (DSV) files, where the delimiter is most commonly a comma or tab, are widely used data format for everything from finance to to-do lists. Mango provides a reader, writer, and formatter for DSV which is configurable to match most standards. The following code snippet illustrates how easy it is to read in a CSV file:

The CSV format can be specified using fluent accessors on the CSV class. A generic CSV and TSV format are accessible via csv() and tsv() respectively, but one can also call builder() which will use default values. You can instruct the reader that the first line of a CSV file is the header by calling hasHeader() on your CSV object. Alternatively, you can specify the header if one is not given in the file using header(String…​) or header(List<String>). When a header is specified you can iterate over the file using: CSV.rowMapStream() which will provide Java stream of Map<String,String>.

Mango uses Jackson for handling JSON. Mango provides a utility class Json to perform basic operations, such as serializing an object to a JSon string or to a resource and deserializing Json into an Object. Additionally, Mango provides a JsonEntry class that wraps the Jackson Json classes allowing easy builder style approaches to constructing Json.

The builder pattern is an excellent way to create Objects which have multiple parameters. However, the builder pattern doesn’t help when we want to specify parameters via configuration or in a concise manner. That is where Mango Specificatons come in handy. A specification is a URI-like object defining a Schema, Protocol, SubProtocols, Path, and Query Parameters that define a resource, connection, etc. The specification form is as follows:

An example is kv:mem:people which defines an in-memory ke-value store with the namespace people. The specifciation kv:disk:people::~/people.db;readOnly=true defines a disk-based key-value store with the namespace people stored at ~/people.db and being accessed as read only. Note that the Path and Query Arguments can will be resolved against the current Config allowing for dynamic paths like ${BASE_DIR}/myFile for paths and parameter=${parameter.defaultValue} where ${BASE_DIR} and ${parameter.defaultValue} will be set via the Config.

Dynamic enumerations are an enum-like objects that can have elements defined at runtime. Elements on a dynamic enumeration are singleton objects. In most cases it is acceptable to use the == operator for checking equality. There are two types of dynamic enumerations:

Both flat and hierarchical enums are uniquely defined by the label used to make them. Labels are restricted to only containing letters, digits, and underscores. Further, all labels are normalized to uppercase. Note that all labels should be unique within the dynamic enumeration.

Dynamic enumeration elements implement the Tag interface, which defines the name(), label(), and isInstance(Tag) methods. For flat enum elements these methods are all based on its normalized label, i.e. name() and label() return the normalized label and isInstance(Tag) checks that the given tag is of the same class and then checks for label name equality. However, hierarchical enum elements are defined with a label and a parent. Therefore, the name() method of hierarchical enum elements returns the full path from the ROOT (but not including the ROOT), e.g. if we have an element with label ScienceTeacher whose parent is Teacher which has ROOT as the parent, the name would be Teacher$ScienceTeacher. The isInstance(Tag) method will travese the hierarchy, such that the method would return true if we ask if Teacher$ScienceTeacher is an instance of Teacher.

Parameter maps are specialized maps that have predefined set of keys (parameters) where each key has an associated type and default value. They are useful to simulate "named and default parameters" found in other languages like Python. However, parameters defined in a parameter map are typed and will validate valeus of the correct type are being assigned. Parameter maps are implemented using the ParamMap class.

In order to define a ParamMap, you must first define the parameters. The first step is to construct a parameter definition (ParameterDef) that maps a parameter name to a type. Parameter definitions can be used by multiple ParamMap. To construct a ParameterDef, we use one of the static methods as such:

With the parameters defined, we can now create a parameter map. Typically, you will want to subclass the ParamMap class setting its generic type to the class you are creating. You will want to define a set of public final variables of type Parameter that will map a parameter definition to a value. Each of the parameters has a default value associated with it, such that whenever the parameter map is used the calling method can be assured that a reasonable value for a parameter will be set. The following example illustrates the definition of a MyParameters parameter map with two parameters.

Now we can define methods that utilize our MyParameters class. We can define the method to take a MyParameters object or to take a Consumer. Examples of this are as follows:

ParamMap have fluent accessors, so that we when using them as the argument to myMethod, we can do the following:

We can also use the public fields directly:

The myMethod2 illustrates how we can mimic named parameters using `Consumer`s. Whe can call the method in the following manner:

In addition to using the public variable, we can also set a parameter’s value using its name as follows:

You can use inheritance to specialize your parameter maps, for example:

Creates an abstract base parameter class (BaseParameters) which defines common parameters (iterations). Child classes (ClusterParameters and ClassifierParameters) then can add parameters specific to their use case. We can then construct a method which takes the BaseParameters, e.g. train(BaseParameters<?> parameters) which we during invocation we can send the correct set of parameters.

When using the BaseParameters class we can cast the class to the correct instance type (e.g. ClassifierParameters) as shown in option 1 or use the getOrDefault methods on the ParamMap as shown in option2.

A tuple is a finite sequence of items. Mango provides specific implementations for degree 0-4 tuples, which all each element’s type to be defined via generics. For tuples with degree 5 or more, a generic NTuple is provided.

Mango provides a posix-like command line parser that is capable of handling non-specified arguments. Command line arguments can be specified manually adding by adding a NamedOption via the addOption(NamedOption) method or automatically based on fields with @Option annotations by setting the parser’s owner object via the constructor. The parser accepts long (e.g. --longOption) and short (e.g. -s) arguments. Multiple short (e.g. single character) arguments can be specified at one time (e.g. -xzf would set the x, z, and f options to true). Short arguments may have values (e.g. -f FILENAME). Long arguments whose values are not defined as being boolean require their value to be set. Boolean valued long arguments can specified without the true/false value. All parsers will have help (-h or --help), config (--config), and explain config (--config-explain) options added automatically.>

Values for options will be specified on the corresponding NamedOption instance. The value can be retrieved either directly from the NamedOption or by using the get(String) method. Argument names need not specify the -- or - prefix.

An example of manually building a CommandLineParser is listed below:

An example of using fields to define your command line arguments is as follows:

Note: All command line arugments specified using as long or short options are automatically put into the Configuration object.

Mango provides a convenient to use Configuration system that allows for the definition of properties and beans. Configuration files have a .conf extension and packages can define a default.conf for its configuration. Mango’s application framework will take care of initializing the configuration for you.

However, if you are not using the application framework you can initialize the Configuration yourself using one of the initialize methods. The initialize methods take some combination of the name of the program being ran, the array of command line arguments, the command line parser, and an optional array of other packages whose configuration settings we need to load. The signature of the full initialize method is as follows:

Mango also provides an initialize method which construct a command line parser (this is useful if you do not care about the command line arguments). Finally, for unit testing there is an initializeTest() method.

Mango will attempt to load a package configuration based on the call class. For all package configurations, Mango will traverse the package tree up until a default.conf is found or it has reaced the root. For example, if a default.conf is defined in com.mycompany and our Config is initialized in a class in the package com.mycompany.apps, it will attempt to resolve the package configuration in the following order:

You can also pass a custom configuration file via the --config command line option. This configuration file will take precedence overriding any values in the default.conf files.

The Config object can be accessed via its static methods. To retrieve values you use a get method which returns a Val object. The following get methods are defined:

The get method calls the findKey method to determine the best matching property name for the given prefix / class name and components.

If the first property component is a Language, it will be used to find language dependent values, if available. The language will be checked using its enum name (e.g. ENGLISH), its lower-cased name (e.g. english), its two-letter language code (e.g. EN), and the lower-cased version of its two-letter language code (e.g. en). When only a language is given a property component it will check for each language variation and if none are found fallback to the just the prefix or class name. For example, given the following configuration:

if we call get("timeout", Language.SPANISH), its call to findKey will return timeout as the prefix exists, but there is not a Spanish specific version. In contrast calling get("timeout", Language.ENGLISH) would result in timeout.en as an English specific value is present. Similarly, when one or more non-Language components are specified the search will be performed looking for prefix.language.components, prefix.components.language, and fallback to prefix.components

Once a property name is found its value will be searched for in the Config object, then in the system properties, and finally as an environment variable.

Mango provides a Preload annotation for classes which will force the static fields to be initialized. This is automatically done as part of Config initialization.