Using ProSort in Various Applications

This chapter describes how ProSort can be used in various applications.

1. ProSort Command

The prosort command can be used in systems where ProSort is installed. Before using the command, environment variables must be set appropriately.

The usage of the prosort command is as follows:

prosort [-h] [-v] [-s] [-j] [script file]
Option Description

-h

Shows brief help for prosort command.

-v

Shows ProSort version.

-s

Shows the following status information after executing the script file.

  • Total Infile Size: input file size

  • Total Allocated Data Block Size: total size of data blocks that save all records

  • Total Allocated Key Block Size: total size of key blocks that save sorting key fields

  • Total Allocated Key Store Size: total size of key stores that save key fields' prefixes

  • Maximum Allocated Memory Size: maximum memory usage

  • Total Disk Read Size: size of data read from disks

  • Total Disk Write Size: size of data written to disks

  • Current Data Block Size: unprocessed data block size

  • Current Key Block Size: unprocessed key block size

  • Current Key Store Size: unprocessed key store size

-j

Shows profile information after executing the script file.

Since profile information varies depending on the product version, this guide does not explain it.

script file

Script file directory path and name. If omitted, a standard input script file is used.

2. ProSort API Functions

ProSort provides the following C API functions. All functions return a value of 0 upon success, and return a negative value upon failure.

Function Description

prosort_run_script

Performs all operations in a script.

prosort_setup

Initializes the data required to execute ProSort.

prosort_setup_with_userexit

Has the same input/output parameters as prosort_setup.

prosort_release_record

Inputs data by record for a sort operation.

prosort_release_end

Ends data input for a sort operation.

prosort_return_record

Gets an operation result by record.

prosort_end

Returns all resources used for operations.

prosort_get_error_number

Gets an error number.

prosort_get_error_message

Gets an error message.

2.1. prosort_run_script

Performs all operations in a script.

  • Prototype

    int prosort_run_script(const char *const script);

  • Parameter

    Parameter Description

    script

    ProSort script used for operations.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

int
main(int argc, char **argv)
{
    int rc;
    char *script =
        "DEFREC FIXED,SIZE=256 "
        "MEMORY 500M "
        "INFILE = (x1,100M, x2, 100M, x3, 100M, x4, 100M) "
        "SORT FIELDS=(1,13,AC,A) "
        "OUTFIL FNAMES = xresult ";

    rc = prosort_run_script (script);
    if (rc < 0) {
        fprintf(stderr,"Error[%d]:\n", rc);
        fprintf(stderr,"%s\n", ps_err_code_to_cause(rc));
        fprintf(stderr,"%s\n", ps_err_code_to_action(rc));

        exit (rc);
    }

    return 0;
}

2.2. prosort_setup

Initializes the data required to execute ProSort. If prosort_run_script is used, prosort_setup does not need to be called.

  • Prototype

    typedef int (*callback_func_t)(char **data_buf_addr,
                               int *data_buf_length);

int prosort_setup (const char *prosort_script,
                   callback_func_t *data_fetch_callback_funcs,
                   int data_fetch_callback_func_cnt,
                   void **ptr_prosort_context);

  • Parameter

    Parameter Description

    *prosort_script

    Pointer to a script used by ProSort APIs.

    *data_fetch_callback_ funcs

    Pointer to a callback function or its array.

    For merge or copy operations, the necessary data is inputted through a callback function. For merge operations, the data can be inputted through multiple callback functions.

    data_fetch_callback_ func_cnt

    Number of callback functions.

    **ptr_prosort_context

    Pointer to context information required to perform operations. The created context can be applied to another API for operations.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

static void
print_err_msg(int ec)
{
    fprintf(stderr,"Error[%d]:\n", ec);
    fprintf(stderr,"%s\n", ps_err_code_to_cause(ec));
    fprintf(stderr,"%s\n", ps_err_code_to_action(ec));
    fprintf(stderr,"%s\n", ps_err_get_last_msg());
}

/* 16-byte record */
#define REC_LEN 16

#define RAND (char) (rand() % 24) + 'A'
int
main(int argc, char **argv)
{
    int   i      = 0;
    int   j      = 0;
    int   rc     = 0;

    void *ps_ctx;

    char buf[REC_LEN];
    const char *script = "MEMORY 100M\n"
                         "DATASIZE 100M\n"
                         "DEFREC FIXED, SIZE=16\n"
                         "SORT FIELDS = (1,8,CH,A)\n";

    /* script setting */
    rc = prosort_setup (script, NULL, 0, &ps_ctx);

    if (rc < 0) {
        print_err_msg (rc);
        exit (rc);
    }

    /* record input */
    for (i = 0; i < 100; i++) {

        for (j = 0; j < REC_LEN - 1; j++)
            buf[j] = RAND;
        buf[j] = 0;

        rc = prosort_release_record (ps_ctx, buf, REC_LEN);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
    }
    /* record input completed */
    prosort_release_end (ps_ctx);

    do {
        char outbuf[REC_LEN];
        char *outbufs[1];
        unsigned int len = REC_LEN;

        outbufs[0] = outbuf;

        rc = prosort_return_record (ps_ctx, outbufs, &len);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
        if (len == 0)
            break;
        printf ("%s\n", outbuf);

    } while (1);

    prosort_end (ps_ctx);
    return 0;
}

2.3. prosort_setup_with_userexit

Has the same input/output parameters as prosort_setup except for func_e15, func_e32, func_e35, and func_e61.

  • Prototype

    typedef int (*user_exit_func_t)(const void *rec_addr,
                                const int rec_len,
                                int file_no,
                                void **ret_rec_addr,
                                int *ret_rec_len);

int prosort_setup_with_userexit(const char *prosort_script,
                                callback_func_t *data_fetch_callback_funcs,
                                int data_fetch_callback_func_cnt,
                                user_exit_func_t func_e15,
                                user_exit_func_t func_e32,
                                user_exit_func_t func_e35,
                                user_exit_func_t func_e61,
                                void **ptr_prosort_context);

  • Parameter

    Parameter Description

    *prosort_script

    Pointer to a script used by ProSort APIs.

    *data_fetch_callback_ funcs

    Pointer to a callback function or its array.

    For merge or copy operations, the necessary data is inputted through a callback function. For merge operations, the data can be inputted through multiple callback functions.

    data_fetch_callback_ func_cnt

    Number of callback functions.

    func_e15

    User Exit Function E15.

    func_e32

    User Exit Function E32.

    func_e35

    User Exit Function E35.

    func_e61

    User Exit Function E61.

    **ptr_prosort_context

    Pointer to the context information required to perform operations. The created context can be applied to another API for operations.

  • Example

    int func_e15(const void *rec_addr,
             const int rec_len,
             int file_no,
             void **ret_rec_addr,
             int *ret_rec_len)
{
    ...
}

int main(int argc, char **argv)
{
    int rc;
    void *ps_ctx;
    const char *script =
        "DEFREC FIXED,SIZE=150 \n"
        "DATASIZE 2M \n"
        "MEMORY 512M \n"
        "WORKSPACE = (./) \n"
        "SORT FIELDS=(1,2,A),FORMAT=BI \n"
        "INCLUDE COND=(1,2,CH,EQ,C'12',OR,1,2,CH,EQ,C'22',OR,1,2,CH,EQ,C'32') \n"
        "OUTFIL FNAMES=OUT1 \n";

    rc = prosort_setup_with_userexit(script, NULL, 0, func_e15, NULL,
                                     NULL, NULL, &ps_ctx);
    ...
}

2.4. prosort_release_record

Inputs data by record for sort operations.

  • Prototype

    int prosort_release_record(void *prosort_context,
                         const char *record,
                         const unsigned int record_length);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

    *record

    Pointer to a record.

    record_length

    Record length.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

static void
print_err_msg(int ec)
{
    fprintf(stderr,"Error[%d]:\n", ec);
    fprintf(stderr,"%s\n", ps_err_code_to_cause(ec));
    fprintf(stderr,"%s\n", ps_err_code_to_action(ec));
    fprintf(stderr,"%s\n", ps_err_get_last_msg());
}

/* 16-byte record */
#define REC_LEN 16

#define RAND (char) (rand() % 24) + 'A'
int
main(int argc, char **argv)
{
    int   i      = 0;
    int   j      = 0;
    int   rc     = 0;

    void *ps_ctx;

    char buf[REC_LEN];
    const char *script = "MEMORY 100M\n"
                         "DATASIZE 100M\n"
                         "DEFREC FIXED, SIZE=16\n"
                         "SORT FIELDS = (1,8,CH,A)\n";

    /* script setting */
    rc = prosort_setup (script, NULL, 0, &ps_ctx);

    if (rc < 0) {
        print_err_msg (rc);
        exit (rc);
    }

    /* record input */
    for (i = 0; i < 100; i++) {

        for (j = 0; j < REC_LEN - 1; j++)
            buf[j] = RAND;
        buf[j] = 0;

        rc = prosort_release_record (ps_ctx, buf, REC_LEN);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
    }
    /* record input completed */
    prosort_release_end (ps_ctx);

    do {
        char outbuf[REC_LEN];
        char *outbufs[1];
        unsigned int len = REC_LEN;

        outbufs[0] = outbuf;

        rc = prosort_return_record (ps_ctx, outbufs, &len);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
        if (len == 0)
            break;
        printf ("%s\n", outbuf);

    } while (1);

    prosort_end (ps_ctx);
    return 0;
}

2.5. prosort_release_end

Ends data inputs for sort operations.

  • Prototype

    int prosort_release_end(void *prosort_context);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

static void
print_err_msg(int ec)
{
    fprintf(stderr,"Error[%d]:\n", ec);
    fprintf(stderr,"%s\n", ps_err_code_to_cause(ec));
    fprintf(stderr,"%s\n", ps_err_code_to_action(ec));
    fprintf(stderr,"%s\n", ps_err_get_last_msg());
}

/* 16-byte record */
#define REC_LEN 16

#define RAND (char) (rand() % 24) + 'A'
int
main(int argc, char **argv)
{
    int   i      = 0;
    int   j      = 0;
    int   rc     = 0;

    void *ps_ctx;

    char buf[REC_LEN];
    const char *script = "MEMORY 100M\n"
                         "DATASIZE 100M\n"
                         "DEFREC FIXED, SIZE=16\n"
                         "SORT FIELDS = (1,8,CH,A)\n";

    /* script setting */
    rc = prosort_setup (script, NULL, 0, &ps_ctx);

    if (rc < 0) {
        print_err_msg (rc);
        exit (rc);
    }

    /* record input */
    for (i = 0; i < 100; i++) {

        for (j = 0; j < REC_LEN - 1; j++)
            buf[j] = RAND;
        buf[j] = 0;

        rc = prosort_release_record (ps_ctx, buf, REC_LEN);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
    }
    /* record input completed */
    prosort_release_end (ps_ctx);

    do {
        char outbuf[REC_LEN];
        char *outbufs[1];
        unsigned int len = REC_LEN;

        outbufs[0] = outbuf;

        rc = prosort_return_record (ps_ctx, outbufs, &len);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
        if (len == 0)
            break;
        printf ("%s\n", outbuf);

    } while (1);

    prosort_end (ps_ctx);
    return 0;
}

2.6. prosort_return_record

Gets an operation result by record. Since there can be multiple outputs because of OUTFIL, the result is obtained by using a buffer array. The array[0] is SORTOUT (output without an OUTFIL execution), and OUTFIL execution results are outputted in the order as described in a script.

  • Prototype

    int prosort_return_record(void *prosort_context,
                        char **output_buffer,
                        unsigned int *output_buffer_length);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

    **output_buffer

    Pointer to the array of addresses of buffers to get output records.

    *output_buffer_length

    Pointer to the array to save output record length.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

static void
print_err_msg(int ec)
{
    fprintf(stderr,"Error[%d]:\n", ec);
    fprintf(stderr,"%s\n", ps_err_code_to_cause(ec));
    fprintf(stderr,"%s\n", ps_err_code_to_action(ec));
    fprintf(stderr,"%s\n", ps_err_get_last_msg());
}

/* 16-byte record */
#define REC_LEN 16

#define RAND (char) (rand() % 24) + 'A'
int
main(int argc, char **argv)
{
    int   i      = 0;
    int   j      = 0;
    int   rc     = 0;

    void *ps_ctx;

    char buf[REC_LEN];
    const char *script = "MEMORY 100M\n"
                         "DATASIZE 100M\n"
                         "DEFREC FIXED, SIZE=16\n"
                         "SORT FIELDS = (1,8,CH,A)\n";

    /* script setting */
    rc = prosort_setup (script, NULL, 0, &ps_ctx);

    if (rc < 0) {
        print_err_msg (rc);
        exit (rc);
    }

    /* record input */
    for (i = 0; i < 100; i++) {

        for (j = 0; j < REC_LEN - 1; j++)
            buf[j] = RAND;
        buf[j] = 0;

        rc = prosort_release_record (ps_ctx, buf, REC_LEN);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
    }
    /* record input completed */
    prosort_release_end (ps_ctx);

    do {
        char outbuf[REC_LEN];
        char *outbufs[1];
        unsigned int len = REC_LEN;

        outbufs[0] = outbuf;

        rc = prosort_return_record (ps_ctx, outbufs, &len);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
        if (len == 0)
            break;
        printf ("%s\n", outbuf);

    } while (1);

    prosort_end (ps_ctx);
    return 0;
}

2.7. prosort_end

Returns all resources used for operations.

  • Prototype

    int prosort_end(void *prosort_context);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

  • Example

    #include <stdio.h>
#include <stdlib.h>
#include "prosort.h"

static void
print_err_msg(int ec)
{
    fprintf(stderr,"Error[%d]:\n", ec);
    fprintf(stderr,"%s\n", ps_err_code_to_cause(ec));
    fprintf(stderr,"%s\n", ps_err_code_to_action(ec));
    fprintf(stderr,"%s\n", ps_err_get_last_msg());
}

/* 16-byte record */
#define REC_LEN 16

#define RAND (char) (rand() % 24) + 'A'
int
main(int argc, char **argv)
{
    int   i      = 0;
    int   j      = 0;
    int   rc     = 0;

    void *ps_ctx;

    char buf[REC_LEN];
    const char *script = "MEMORY 100M\n"
                         "DATASIZE 100M\n"
                         "DEFREC FIXED, SIZE=16\n"
                         "SORT FIELDS = (1,8,CH,A)\n";

    /* script setting */
    rc = prosort_setup (script, NULL, 0, &ps_ctx);

    if (rc < 0) {
        print_err_msg (rc);
        exit (rc);
    }

    /* record input */
    for (i = 0; i < 100; i++) {

        for (j = 0; j < REC_LEN - 1; j++)
            buf[j] = RAND;
        buf[j] = 0;

        rc = prosort_release_record (ps_ctx, buf, REC_LEN);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
    }
    /* record input completed */
    prosort_release_end (ps_ctx);

    do {
        char outbuf[REC_LEN];
        char *outbufs[1];
        unsigned int len = REC_LEN;

        outbufs[0] = outbuf;

        rc = prosort_return_record (ps_ctx, outbufs, &len);
        if (rc < 0) {
            print_err_msg (rc);
            exit (rc);
        }
        if (len == 0)
            break;
        printf ("%s\n", outbuf);

    } while (1);

    prosort_end (ps_ctx);
    return 0;
}

2.8. prosort_get_error_number

Gets an error number.

  • Prototype

    int prosort_get_error_number(void *prosort_context);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

  • Example

    int main(int argc, char **argv)
{
    int rc;
    void *ps_ctx;
    const char *script =
        "DEFREC FIXED,SIZE=150 \n"
        "DATASIZE 2M \n"
        "MEMORY 512M \n"
        "WORKSPACE = (./) \n"
        "SORT FIELDS=(1,2,A),FORMAT=BI \n"
        "INCLUDE COND=(1,2,CH,EQ,C'12',OR,1,2,CH,EQ,C'22',OR,1,2,CH,EQ,C'32') \n"
        "OUTFIL FNAMES=OUT1 \n";

    rc = prosort_setup(script, NULL, 0, &ps_ctx);
    if(rc < 0) {
        fprintf(stderr, "Error number is %d\n", prosort_get_error_number(ps_ctx));
    }
    ...
}

2.9. prosort_get_error_message

Gets an error message.

  • Prototype

    int prosort_get_error_message(void *prosort_context,
                            char *error_message_buffer,
                            unsigned int *error_message_length);

  • Parameter

    Parameter Description

    *prosort_context

    Pointer to a context created in prosort_setup.

    *error_message_buffer

    Pointer to the buffer for getting an error message.

    *error_message_length

    Pointer to the error message length.

  • Example

    #define ERR_MSG_MAXLEN 1024
int main(int argc, char **argv)
{
    int rc;
    void *ps_ctx;
    char *err_msg;
    unsigned int err_msg_len;
    const char *script =
        "DEFREC FIXED,SIZE=150 \n"
        "DATASIZE 2M \n"
        "MEMORY 512M \n"
        "WORKSPACE = (./) \n"
        "SORT FIELDS=(1,2,A),FORMAT=BI \n"
        "INCLUDE COND=(1,2,CH,EQ,C'12',OR,1,2,CH,EQ,C'22',OR,1,2,CH,EQ,C'32') \n"
        "OUTFIL FNAMES=OUT1 \n";
    err_msg = (char *)malloc(sizeof(char) * ERR_MSG_MAXLEN);
    err_msg_len = ERR_MSG_MAXLEN;

    rc = prosort_setup(script, NULL, 0, &ps_ctx);
    if(rc < 0) {
        prosort_get_error_message(ps_ctx, err_msg, &err_msg_len);
        fprintf(stderr, "Error message is %s\n", err_msg);
    }
    ...
}

3. Starting ProSort in OpenFrame/Batch

OpenFrame uses DFSORT to create an output dataset by sorting, merging, copying, filtering, editing, or adding up input datasets through ProSort functions.

To perform sorting in OpenFrame, job control statements must be written in Job Control Language (JCL) and sent to OpenFrame/Batch as a batch job (JOB).

For more information about batch jobs, refer to OpenFrame Batch Guide and OpenFrame TJES Guide.

3.1. Job Control Statements

Job control statements can consist of JOB, EXEC, DD, /*, and other statements.

For more information about JCL, refer to OpenFrame JCL Reference Guide.

  • JOB statement

    Indicates the start of a JOB and defines the JOB’s attribute.

    The following is an example.

    //SORT00  JOB  CLASS=A,MSGCLASS=X,MSGLEVEL=(1,1)

  • EXEC statement

    One JOB consists of multiple STEPs. This statement indicates the start of a STEP and specifies a program name and parameters to be sent to the program. To perform sorting by using ProSort, PGM=DFSORT must be specified.

    The following is an example.

    //SORT  EXEC PGM=DFSORT,REGION=2M

  • DD statement

    Specifies the name and attribute of a dataset to sort by using the following items.

    Item Description

    SYSIN

    Dataset used to input a SORT control statement.

    Typically, a system input stream is used as DD *. SORT control statements must be written based on DFSORT control statements, not ProSort control statements.

    SYSOUT

    Dataset used to list control statements and output messages.

    Typically, system output stream is used.

    SORTIN

    Input dataset to sort or copy.

    SORTINnn

    Input dataset to merge. Dataset names do not need to be defined in order, from SORTIN01, but cannot be a duplicate.

    SORTINWKnn

    Dataset used as a work area while sorting.

    SORTOUT

    Dataset used to output a sort, merge, or copy result.

    If SORTIN or SORTINnn is fixed-length (FB), SORTOUT is also fixed-length. If SORTIN or SORTINnn is variable-length (VB), SORTOUT is also variable-length. NVSM input records can be outputted as VSAM records, and vice versa.

    SORTOFxx

    Dataset used to output multiple datasets when records are reformatted with OUTFIL. xx consists of one or two alphabetic letters or digits.

    SYMNAMES

    Dataset referenced when symbols are substituted.

    • Record format

      symbol name, field start position, field length, field format

    JNFnCNTL

    Control statement for subtask n when performing a JOIN. Defines the SORT statement to be executed before the JOIN operation. n is either 1 or 2.

  • /* statement

    Indicates the end of a SYSIN dataset. The end can also be indicated by using END in a SORT control statement.

3.2. Dataset Definition

The following are operands used to define datasets.

Dataset DCB DISP DSN LABEL SPACE UNIT VOL AMP

SORTIN

1

O

O

2

X

3

3

7

SORTINnn

1

O

O

2

5

3

3

7

SORTOUT

1

O

4

2

5

3

6

7

SORTOFxx

1

O

4

2

5

3

6

7

SORTWKnn

X

4

4

X

5

3

6

X

JNFnCNTL

1

4

4

2

5

3

3

7

  • O: must be specified.

  • X: does not need to be specified.

  • 1: needs to be specified in some cases.

  • 2: unnecessary on disks.

  • 3: must be specified unless the dataset is written or cataloged before a current JOB STEP.

  • 4: must be specified for non-temporary datasets.

  • 5: must be specified to write on a direct access device.

  • 6: must be specified unless the dataset is temporary, it is written and cataloged before a current STEP, or it is passed from another STEP in the same JOB.

  • 7: can be specified for VSAM datasets, but unnecessary for other datasets.

DCB Operand’s Subparameters

For non-VSAM input/oputput datasets, the following items must be specified.

Item Description

Record format (RECFM)

Format of records in a dataset.

Record length (LRECL)

Maximum logical length of a record in a dataset. (Unit: bytes)

Block size (BLKSIZE)

Maximum physical length of a record in a dataset. (Unit: bytes)

The items are determined in the following order.

  1. If RECFM, LRECL, and BLKSIZE are specified in a DD statement, the information is used and others are ignored.

  2. If DCB operand is not specified for an input dataset, an error occurs. However, if a dataset already exists, the existing information is used. If DCB operand is not specified for an output dataset, the information is determined in the following order.

    1. Information specified in OUTREC is used.

    2. Information specified in RECORD is used.

    3. Information specified in INREC is used.

    4. Information specified in an input dataset is used.