Code review or Technology Knowhow – Getting it right

Coding is considered the most simplest of the tasks in a software development life cycle. Most people I have seen want to get the code right the first time they write, a bang for buck for the time spend. If only it works that wayJ. But the interesting part is that good code is hard to find. It’s not quite too often that I hear the words – “We should corner the guy who wrote this and beat the hell out of him”. So what makes people write bad code? Just a code review that went horribly wrong or the programmers coding concept which is wrong or a cheat sheet of best practices in coding which is way too wrong? The blame as it stands invariably shifts among all the three and comes a full circle back to the developer.

In my earlier posts I have mentioned instances where we were able to optimize code to such an extent that, it cut down cpu and run time of a process that the result send shockwaves through the developer community in the firm – A process which was running for 1.5 hrs printing 6 records in the output and consuming a good load of cpu a year began running in less than a minute and using cpu that would have last another 30 years if it ran every day as compared to its usage for 1 day now.

Well that puts us in a spot. And that brings to question, what’s a good coding practice? Or is there definite guidelines that one need to follow? Or better still, how good do you know your technology?

A few thumb rules go a long way:

·         Know your data

·         Make sure you have a design (either document it or have it in your mind)

·         Run the design through your team or peers (however small it maybe)

·         If you are planning to use any jazzy feature see how good it performs in the system or better still check with someone who has used it

·         And if possible get a code review with an expert in technology

Dynamic Variable with Dynamic Array Occurrence

Most of has used dynamic variables in natural, with 4.1 we can abuse it further by utilizing it in dynamic array as well. Probably I should have explained what came along with 4.1 features first but one of the biggest advantages of 4.1 features is its advantage to hold more than 32K of data in work area. In fact we can have 1GB (but please check with your environment support or DBA’s before you decide to invoke these features, it may snap your system if not wisely coded). The major advantage of dynamic variable with dynamic array occurrence is the elimination of “Index not within array structure” errors since we code it in such a way as to have a upper limit based on forecasted growth. And if used wisely, we can eliminate this error completely (but please make sure someone does a good code reviewJ). The option is a double edged sword: it not only eliminates “Index not within array structure” and loads or dynamically expands array data based on growth but with the negative fallout that if it exceeds 1GB your process falls over. So be very cautious on its usage.

The simple trick in a dynamic variable with dynamic array occurrence is to capture the current occurrence by just moving the *COUNTER variable to the counter variable to expand the dynamic variable & array. The result is – a characteristic of passed field is taken as characteristics of the result field for dynamic field. And it’s most efficient usage would be to use in loading reference-data.

A sample piece of code will look like this:

DEFINE DATA LOCAL 01 REF VIEW OF REFERENCE-DATA   02 RD-KEY             /* 15 BYTES OF SUPER-DESCRIPTOR   02 REDEFINE RD-KEY      03 RD-KEY1 (A5)          /* FIRST HALF OF SUPER      03 RD-DATA (A10)         /* SECOND HALF OF SUPER 01 #START-RD-KEY (A15) 01 #END-RD-KEY   (A15) 01 #CITY-ARRAY   (A/*) DYNAMIC 01 #CITY-COUNT   (I4) 01 #CITY-INDEX   (I4) END-DEFINE * RESET #START-RD-KEY #END-RD-KEY #CITY-COUNT COMPRESS 'MIAMI' INTO #START-RD-KEY LEAVING NO SPACE COMPRESS 'MIAMI' H'FF' INTO #END-RD-KEY LEAVING NO SPACE * HISTOGRAM MULTI-FETCH OF 1000 REF FROM #START-RD-KEY  TO #END-RD-KEY   ADD 1 TO #CITY-COUNT /* Or Move *COUNTER   EXPAND ARRAY #CITY-ARRAY TO (1:#CITY-COUNT) /* expands the array to counter   MOVE RD-DATA TO #CITY-ARRAY (#CITY-COUNT) /* resultant variable takes characteristics of moving variable END-HISTOGRAM * * Sample code to check if data is loaded IF #CITY-COUNT > 0   IF #CITY-ARRAY (*) EQ SCAN 'MIAMI' /* SCAN is a new option in 4.1 and similar to examine but just checks for existence of a value       WRITE 'MATCH FOUND FOR MIAMI'   ELSE       WRITE 'MATCH NOT FOUDN FOR MIAMI'   END-IF END-IF END

If used wisely the advantages are limitless providing a good optimization in many fronts wherever applicable, probably opening the Pandora’s Box for another discussion (loading most frequent data to array for optimized performance :-)).

Nitty Gritty ways of Optimization

Unlike the conventional ways of optimization, there are few minute ways of writing a good code. A simple practice if followed would help write an optimized code to a good potential.

READ WORK FILE NN RECORD #FILE

Well a READ WORK FILE statement needs no introduction. But it just comes down to the keyword used along with it. In most of the 2.2/3.1 codes, we have just used it by the multiple layout structure in conjunction like:

READ WORK FILE 1 #A(A250) #B(A250) #C(A250) #D(A250)

Hmm, giving it a little thought, it can be optimized. In 3.1 we could just optimize using the RECORD clause and then RDEFINING a GROUP variable over the layout, like:

1 #FILE-IN

1 REDEFINE #FILE-IN

2 #A(A250)

2 #B(A250)

2 #C(A250)

2 #D(A250)

And then changing the READ WORK FILE statement as below:

READ WORK FILE 1 RECORD #FILE-IN

Well what we have accomplished by the above statement is to READ the layout as one single layout of 1000 bytes as opposed to READing it through multiple layouts each redefined alphanumeric field of 250 bytes each. The simple performance issue with “READ WORK FILE 1 #A(A250) #B(A250) #C(A250) #D(A250)” is that it reads each layout and validates each of the variables inside each of the layouts and then proceeds to the next statement within the program. So, we end up validating all fields redefined inside #A, #B, #C & #D and thus increasing the run time and cpu for the process, what we fail to realize is that a simple RECORD statement will optimize it to such a big extend that if the number of records are huge, we end up saving a lot.

Well a 4.1, code is much better, we just need to change the statement to

READ WORK FILE 1 RECORD #FILE-IN (A1000)

Similar, except that instead of defining it as a group field, we can define it as a single field of 1000 bytes! Yes coming to reality if you haven’t explored the features of Natural 4.1, we can have alphanumeric data upto 1GB. So similarly we end up reading it as a 1000 byte of single alphanumeric field and validate it as a single field of alphanumeric field as opposed to fields redefined inside it.

RESTOW TO HIGHER VERSION

Well this is plain simple RESTOW; just get all your code to the next higher version of the Natural version. The performance improvement of just recompiling code to a 4.2 from a 2.1/2.2 is around 30% reduction in CPU and increased throughput in processing.

  

INLINE PROGRAM STATEMENTS AS OPPOSED TO SUBROUTINE/SUBPROGRAM

Not sure how many of you might agree on this thought, but conceptually it should show improvements if the code is written from top to bottom without any subroutine calls (internal/external) or subprogram calls. The perception that a subroutine is to be used only if it’s being called multiple times has really gone well with the application developer, now we code it just to give it a better readability format! A call to external subroutine/subprogram invokes additional overhead in the system as opposed to coding it inline.

A 2 line statement which gets used multiple times within the program goes to an internal subroutine. Grrr... I hate to say it but ever since I see that kind of code. I feel somebody just followed the book not giving much of a thought!!! I feel a code is more readable if written in one stretch – you don’t need to go down to a subroutine and then come back up to continue with the logic. Consider if we have to go down and back multiple times, how much time would it take to understand such a code. There must be some performance gain (probably in milli seconds) if a code is just one chunk of 1000 lines instead of a code of 500 subroutines with 2 lines each. No offence intended to COBOL programmers, but it’s often seen in their coding styles J.. Either there is no improvement or just venting my frustration :-)

BULK UPDATES, KEEPING UPDATES & EXTRACTION SEPARATE

Mostly this is rule of thumb followed by developers to keep updates and extraction separate. And when in batch to do it using GET & using counter to do bulk updates. This also helps reduce Natural error of NOT holding more records in the buffer than what system can handle.

An efficient code would look something like this:

R1.

READ EMPLOYEES TABLE WITH CITY = ‘MIAMI’ TO ‘MIAMI’

 REJECT IF RESIDENT EQ ‘N’

 GET EMPLOYEE-V2 (R1.)

 CITY: = ‘VEGAS’

 UPDATE EMPLOYEES

 ADD 1 TO #CNT

 IF #CNT GE 25

   RESET #CNT

   END TRANSACTION

 END-IF

END-READ

END TRANSACTION         /* Capturing last record update conditions

USING HISTOGRAM/FIND NUMBER AS OPPOSED FIND/READ TO CHECK IF RECORD EXIST

Sometimes (not too occasionally) we come across programs where people check for exist of record by READ/FIND statements when we have the full value. I feel, a little thought on how Adabas works will help you a lot on when to use READ/FIND vs an optimized option of using HISTOGRAM/FIND NUMBER. Although these statements, and their corresponding Adabas commands, return essentially the same result, how they determine those results differs drastically.

To quote from a text book or theoretical perspective a READ/FIND causes invoking data storage even if you wish to just check existence in inverted list. So a HISTOGRAM/FIND NUMBER is a cheaper call since it restricts your call to just the inverted list.

When to use FIND NUMBER over HISTOGRAM:

If the expected value of *NUMBER is

·         0 to 50 then use FIND NUMBER

·         If *NUMBER will be a large value (upto 1000), then use HISTOGRAM.

·         If *NUMBER will be a very large value, then use READ FROM/TO & ESCAPE BOTTOM.

USING RIGHT SUPERS

Not sure but how many of us have seen people not using the correct super and understanding what data we are retrieving when we write a new code. Well the answer is most of us. I remember in my previous organization we were optimizing most of the batch process and saw an instance where a job was running for 1.5 hours making around 40 million Adabas calls a day and all it was printing was 6 records in the output report!! My colleague was mentioning “We should corner the fellow who wrote this code and beat the hell out of him”. Why? Simple enough, the author of the original program was not using the correct super and with a little bit of work around it was possible to optimize the code so much so that it started completing in 1.5 minutes with around 10 Adabas calls a day.

Well on a lighter note, the optimization effort wouldn’t have been put in place by industry if for the bad code people write. If only... people obey some thumb rules………….

Natural – DB2 ROWSET POSITIONING

Its multiple Row processing in Natural-Db2 process which helps reduce cpu, increase throughput and save MIPS. This is in some way similar to Natural-Adabas MULTI-FETCH  parameter option where in we specify the multiple row selection when Natural issues the call to Adabas and the limitation of using 32K in view size or fetched record size.

Advantages of using the option

• Improved throughput through fewer database access calls and lower network operations
• Lower CPU consumption

So the Purpose of Rowset positioning is similar to MULTI-FETCH option in Natural-Adabas, a ROWSET POSITIONING offers to read multiple records with a single database call. At statement execution time, the runtime checks if a multi-fetch-factor greater than 1 is supplied for the database statement and the database call is prepared dynamically to read multiple records with a single database access into the buffer.

Prerequisite

• The option will work only for DB2 version 8 and above only.
• Set the compiler option DB2ARRY=ON (through inline program statement coded after END-DEFINE statement or through NOCOPT command in the Natural editor
• Specify a list of receiving array fields in the INTO clause of the SELECT query. Please note for ROWSET option to work, the length of receiving fields should be same as length of the fields in DB2 table otherwise the process will fall over at the time of execution.
• Specify a variable receiving the number of rows retrieved from the database via the ROWS_RETURNED Clause. Variable has to be defined as I4.
• Specify the number of rows to be retrieved from the database by a single FETCH operation via the WITH ROWSET POSITIONING Clause with a value between 0 & 32767. Variable has to be defined as I4.

Looking at a sample program below:

DEFINE DATA LOCAL                                                  01 NAME            (A20/1:10)                                      01 ADDRESS         (A100/1:10)                                     01 DATEOFBIRTH     (A10/1:10)                                       01 SALARY          (P4.2/1:10)                                     01 L§ADDRESS       (I2/1:10)                                       01 ROWS            (I4)                                            01 NUMBER          (I4)                                            01 INDEX           (I4)                                            END-DEFINE                                                         OPTIONS DB2ARRY=ON                                                 ASSIGN NUMBER := 10                                                SEL.                                                               SELECT NAME, ADDRESS , DATEOFBIRTH, SALARY                                INTO  :NAME(*),                             /* <-- ARRAY                 :ADDRESS(*) LINDICATOR :L§ADDRESS(*), /* <-- ARRAY                 :DATEOFBIRTH(1:10),                   /* <-- ARRAY                 :SALARY(01:10)                        /* <-- ARRAY          FROM EMPLOYEE       WHERE NAME > ' '                                                   WITH ROWSET POSITIONING FOR :NUMBER ROWS     /* <-- ROWS REQ       ROWS_RETURNED :ROWS                          /* <-- ROWS RET   IF ROWS > 0                                                           FOR INDEX = 1 TO ROWS                                        DISPLAY                                                                    INDEX (EM=99) *COUNTER (SEL.) (EM=99) ROWS (EM=99)                 NAME(INDEX)                                                        ADDRESS(INDEX) (AL=20)                                             DATEOFBIRTH(INDEX)                                                 SALARY(INDEX)                                             END-FOR                                                          END-IF                                                           END-SELECT                                                         END

What the above program accomplished is reading 10 rows of data by a single call to EMPLOYEE table. The local field NUMBER helped in telling DB2 to fetch 10 rows satisfying the criteria in the SQL statement. The number of records returned by DB2 is saved in ROWS field.

Optimization – the ever powerful SORTJOIN (for Match and Merge)

Well an advert shift from Natural to SORT, what might have triggered this. Well nothing much but just that mainframe developers tend NOT to use more advanced features and stick to their guns in using program as opposed to the available new features. Well one power packed feature is what SORTJOIN has to offer us. The join facility of SyncSort provides the capability to join records from two source files. Each record from the first file with a given value in one or more fields (the join key) is joined to each record from the second files that has an identical value in its join key, something like a match & merge process. The joined records are passed to the sort/copy process. The power of this facility is enhanced by the ability to eliminate records from either or both files.

The following diagram correctly captures how the SORTJOIN works in its sequence of execution (courtesy SYNSORT manual).

Join processing is controlled by three control statements:

JOINKEYS Enables join feature processing and identifies the fields used to select records for join processing.

JOIN Specifies the disposition of paired and unpaired records in a join.

REFORMAT Defines the record layout to be produced by join processing.

The following examples detail how SORT JOIN can be used:

1. Problem (Using Simple Join with Record selection)

There are two flat files, we are trying to merge the matching data, i.e. by customer number in both the files (There is a 10-digit customer number that is common to the records in both files.) - Names from File 1 & City from the second file. The data required for this task is located in two separate files. The first file contains Name & Customer Number information. The second file contains the Customer Number & City of all the customers. There is a 10-digit customer number that is common to the records in both files.

Input File 1: Transaction File

DCB=(RECFM=FB,LRECL=24)

Position Length Description

1 14 Customer Name

15 10 Customer Number

1---+----0----+----0----+----0----+----0----+ GREG DAVIS    6456332132 MARK LUKE     5435313132 JUDY FOSTER   7343423456 GEORGE STORMS 8933313324 STEVE JONES   2135212331 MEGAN MADRUS  5645321323 MONICA FELDER 3234139897 MARY COX      9893133733 ARTHUR FREEMAN4233232989 JAMES BOND    6462168111

Input File 2: Name & Address File
DCB=(RECFM=FB,LRECL=20)
Position Length Description
1 10 Customer Number
11 10 City

1---+----0----+----0----+----0----+----0----+----0----+----0----+----0 6456332132NEW YORK 5435313132CALIFORNIA 7343423456FLORIDA 8933313321WALES 2135212331VEGAS 5645321323WARWICK 3234139899DETROIT 9893133733JERSEY CITY 4233232989CAROLINA 6462168111VIRGINIA 7833452312HOUSTON 7192837465HAWAI

Approach

JOINKEYS Specifies the location in each file of the field (customer number) used to join the files.  

REFORMAT Defines the record layout to be produced by the join processing.

SORT Specifies the sort key and order (customer number, ascending).

Control Statements

JOINKEYS FILES=F1, FIELDS=(15,10,A), CUSTOMER NUMBER

JOINKEYS FILES=F2, FIELDS=(1,10,A) CUSTOMER NUMBER

REFORMAT FIELDS=(F1:1,24, F2:1,10) NAME, Customer Number & City

SORT FIELDS=(1,10,CH,A)

O/p data as follows

GREG DAVIS    6456332132NEW YORK MARK LUKE     5435313132CALIFORNIA JUDY FOSTER   7343423456FLORIDA STEVE JONES   2135212331VEGAS MEGAN MADRUS  5645321323WARWICK MARY COX      9893133733JERSEY CITY ARTHUR FREEMAN4233232989CAROLINA JAMES BOND    6462168111VIRGINIA

1. Problem (Using Simple Join with UNPAIRED selection)

Using the same example as above, we want to now extract only records in File 1 but not in File 2 for the key (Customer Number)

Control Statements

JOINKEYS FILES=F1, FIELDS=(15,10,A), CUSTOMER NUMBER

JOINKEYS FILES=F2, FIELDS=(1,10,A) CUSTOMER NUMBER

JOIN UNPAIRED,F1,ONLY

SORT FIELDS=(1,10,CH,A)

Approach

JOINKEYS Specifies the location in each file of the field (customer number) used to join the files.  

JOIN Requests only those records that appear in File 1 but not in File2.

SORT Specifies the sort key and order (customer number, ascending).

O/p data as follows:

GEORGE STORMS 8933313324 MONICA FELDER 3234139897

Déjà vu or is the sky really falling down?

Most of us have read about the 2008 crisis of the global melt down and have experienced it first hand as well. Most of the analyst thought it might end up in a “Great Depression” like 1890’s – because the financial system collapsed. It took down a lot of mighty ones - Most of the banks were bailed out and some went down. Well I had my own experience of that as well. Well to tell the truth – it scared the hell out of us, the fear factor – are the dreams coming down? Do we stand any chance in the market now? Lot of questions and most of the people were frantically searching for the answers. But finally it all ended well – at least here in this side of the world.

When financial systems break or there is a perceiving sense that the whole world economy is coming down, all grounds goes loose – not just from jobs scenario, but just the fact everyone suddenly seem noticing change, inflation, equity markets, price of gold and what not. But the simple question is, are we all Socrates or a Nostradamus to predict future? But it does bring to us a lot of information – you start researching on everything related to a meltdown and then you start joining the dots. To tell the truth, when you start doing that you began to believe in the views people express as news.

Well in 2008 I was right in the middle of the storm and my forte was Chapter 11 of Bankruptcy, Mortgage Back Securities & Credit Default Swaps. The vibe that something was wrong was pretty evident at that time, probably because I was in NY @ that time for a short stint @ my old firm and had a first-hand experience of what was happening on the ground before the whole global economy collapsed because of financial system collapses. Probably people knew it was coming but to what extent... An in depth information on the collapse of biggies in Wall Street was captured pretty well in A Colossal failure of Commonsense and Too Big To Fail. Come to think of it, A Colossal failure of Commonsense has been aptly titled if commonsense is tied to ego and perseverance that it will only take a short time to grow big and mighty.

Its 2011, things have relatively changed. We are no longer talking about bankruptcy or MBS or CDS, there are larger issues now - Global economy, Inflation, Crude Prices, PIGS defaults (not the animal but the countries Portugal, Italy, Greece & Spain), Price of Gold, Silver Thursday and full circle back to Double dip recession?. Well now we read what Wikipedia provides, what SIFMA provides and what the street provides and we become an epitome of all evils (yeah too much of knowledge is worse, you add 2 and 2 and say its 5 J). It’s been going rounds for quite some time that Greece may (Stock markets nose dives when it’s swaying to this side) or may not (Flat or slightly higher because there are other Euro zones which can come into effect) default and would need a bulk load of bullion $ to save it from default. And suddenly the whole world is talking about country rating, US downgrade and if India will be re-rated. End result – No change for India.

So just like we have a high frequency trading desk to do analytics based on market conditions and do the correct bets and make money for the firm or instruments which can warn geological or natural catastrophe is there an option to warn a global meltdown or do we go by our Deja vu instincts.. That’s Food for thought or a sword hanging out there?