pegasus.mapreduce.common.utils

Class IOUtils

java.lang.Object

pegasus.mapreduce.common.utils.IOUtils

public class IOUtils
extends java.lang.Object

Constructor Summary

Constructors

Constructor and Description
IOUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	getIntermediateVectorPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, java.lang.String prefix, int blockId) Returns a path of intermediate vector file.
static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.IntWritable,T>	getIntermediateVectorReader(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path path, java.lang.Class<T> valueClass) Opens the intermediate vector file and returns a reader object to read values from the file.
static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.IntWritable,T>	getIntermediateVectorReader(org.apache.hadoop.mapreduce.TaskAttemptContext context, java.lang.String prefix, int blockId, java.lang.Class<T> valueClass) Opens the intermediate vector file and returns a reader object to read values from the file.
static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.IntWritable,T>	getIntermediateVectorWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path path) Opens an intermediate vector file and returns a writer object to write values to the file.
static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.IntWritable,T>	getIntermediateVectorWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context, java.lang.String prefix, int blockId) Opens an intermediate vector file and returns a writer object to write values to the file.
static org.apache.hadoop.fs.Path	getLocalTemporaryPath(java.lang.String name) Returns a path of local temporary file.
static org.apache.hadoop.fs.Path	getLocalTemporaryPath(java.lang.String name, int id) Returns a path of local temporary file.
static org.apache.hadoop.fs.Path	getLocalTemporaryPath(java.lang.String name, org.apache.hadoop.mapreduce.TaskAttemptContext context) Returns a path of local temporary file.
static org.apache.hadoop.io.compress.CompressionCodec	getPreferredCompressionCodec(org.apache.hadoop.conf.Configuration conf) Returns a codec object which is preferred.
static org.apache.hadoop.io.compress.CompressionCodec	getPreferredCompressionCodec(org.apache.hadoop.mapreduce.TaskAttemptContext context) Returns a codec object which is preferred.
static java.io.DataInputStream	getTemporaryFileInputStream(org.apache.hadoop.fs.Path path, org.apache.hadoop.mapreduce.TaskAttemptContext context) Opens the given local temporary file and returns an input stream object to read values from the file.
static java.io.DataOutputStream	getTemporaryFileOutputStream(org.apache.hadoop.fs.Path path, org.apache.hadoop.mapreduce.TaskAttemptContext context) Creates a local temporary file and returns an output stream object to write values to the file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOUtils

public IOUtils()

Method Detail

getIntermediateVectorPath

public static org.apache.hadoop.fs.Path getIntermediateVectorPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                  java.lang.String prefix,
                                                                  int blockId)
                                                           throws java.io.IOException

Returns a path of intermediate vector file.

Parameters:

context - The current execution context

prefix - A prefix string of vector file name

blockId - The id of vector block

Returns:

The path of intermediate vector file

Throws:

java.io.IOException

getIntermediateVectorWriter

public static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.IntWritable,T> getIntermediateVectorWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                                                                                                                 java.lang.String prefix,
                                                                                                                                                                 int blockId)
                                                                                                                                                          throws java.io.IOException

Opens an intermediate vector file and returns a writer object to write values to the file.

Type Parameters:

T - The type parameter of the values to be written

Parameters:

context - The current execution context

prefix - A prefix string of vector file name

blockId - The id of vector block

Returns:

A writer object to write values to the file.

Throws:

java.io.IOException

getIntermediateVectorWriter

public static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.IntWritable,T> getIntermediateVectorWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                                                                                                                 org.apache.hadoop.fs.Path path)
                                                                                                                                                          throws java.io.IOException

Opens an intermediate vector file and returns a writer object to write values to the file.

Type Parameters:

T - The type parameter of the values to be written

Parameters:

context - The current execution context

path - The path of intermediate vector file

Returns:

A writer object to write values to the file.

Throws:

java.io.IOException

getIntermediateVectorReader

public static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.IntWritable,T> getIntermediateVectorReader(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                                                                                                                 java.lang.String prefix,
                                                                                                                                                                 int blockId,
                                                                                                                                                                 java.lang.Class<T> valueClass)
                                                                                                                                                          throws java.io.IOException

Opens the intermediate vector file and returns a reader object to read values from the file.

Type Parameters:

T - The type parameter of the values to be read

Parameters:

context - The current execution context

prefix - A prefix string of vector file name

blockId - The id of vector block

valueClass - A class of the values to be read

Returns:

A reader object to read values from the file

Throws:

java.io.IOException

getIntermediateVectorReader

public static <T extends org.apache.hadoop.io.Writable> org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.IntWritable,T> getIntermediateVectorReader(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                                                                                                                                 org.apache.hadoop.fs.Path path,
                                                                                                                                                                 java.lang.Class<T> valueClass)
                                                                                                                                                          throws java.io.IOException

Opens the intermediate vector file and returns a reader object to read values from the file.

Type Parameters:

T - The type parameter of the values to be read

Parameters:

context - The current execution context

path - The path of intermediate vector file

valueClass - A class of the values to be read

Returns:

A reader object to read values from the file

Throws:

java.io.IOException

getLocalTemporaryPath

public static org.apache.hadoop.fs.Path getLocalTemporaryPath(java.lang.String name,
                                                              org.apache.hadoop.mapreduce.TaskAttemptContext context)

Returns a path of local temporary file.

Parameters:

name - The name of temporary file

context - The current execution context

Returns:

A path of local temporary file

getLocalTemporaryPath

public static org.apache.hadoop.fs.Path getLocalTemporaryPath(java.lang.String name,
                                                              int id)

Returns a path of local temporary file.

Parameters:

name - The name of temporary file

id - A id of temporary file

Returns:

A path of local temporary file

getLocalTemporaryPath

public static org.apache.hadoop.fs.Path getLocalTemporaryPath(java.lang.String name)

Returns a path of local temporary file.

Parameters:

name - A name of temporary file

Returns:

A path of the local temporary file

getTemporaryFileInputStream

public static java.io.DataInputStream getTemporaryFileInputStream(org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                           throws java.io.IOException

Opens the given local temporary file and returns an input stream object to read values from the file.

Parameters:

path - The path of local temporary file

context - The current execution context

Returns:

An input stream object to read values from the file

Throws:

java.io.IOException

getTemporaryFileOutputStream

public static java.io.DataOutputStream getTemporaryFileOutputStream(org.apache.hadoop.fs.Path path,
                                                                    org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                             throws java.io.IOException

Creates a local temporary file and returns an output stream object to write values to the file.

Parameters:

path - A path of local temporary file

context - The current execution context

Returns:

An output stream object to write values to the file

Throws:

java.io.IOException

getPreferredCompressionCodec

public static org.apache.hadoop.io.compress.CompressionCodec getPreferredCompressionCodec(org.apache.hadoop.conf.Configuration conf)

Returns a codec object which is preferred.

Parameters:

conf - The configuration of current execution

Returns:

The codec object which is preferred

getPreferredCompressionCodec

public static org.apache.hadoop.io.compress.CompressionCodec getPreferredCompressionCodec(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Returns a codec object which is preferred.

Parameters:

context - The current execution context

Returns:

The codec object which is preferred