Test Driven Design

My last post was critical of the new control language created to implement a concept called Behaviour Driven Design (BDD).

The process of creating a control language seems liberating at first because the creation process helps improve communications among the creators, but it soon becomes oppressive to those on the outside.

That criticism aside, I actually agree with many of the ideas discussed in Test Driven Design and Behaviour Driven Design.

A quality software shop tests it products thoroughly. Tests are difficult because one needs to test for both expected and unexpected conditions.

Programmers have a tendency to write programs for the way that they imagine people will use their program and fail to realize the creative things people actually do with code. The testers have to test the software for all the different permutations that might happen with the software. This means throwing invalid data at the software and even throwing malicious code at the software to see how it fares against hackers.


Imagine a simple procedure with 5 variables. The testing department has to test all the permutations of those variables.

Object Oriented Programs are even harder to test. Imagine a class with a dozen methods. The testing department needs to test all the paths through that object. 

I find that my testing scripts are generally magnitudes longer than the software itself.

New programs generally need to be tested against existing data. Which is itself a chore.

The idea behind Test Driven Design is that companies should design their tests before writing the software. This seems odd at first. But on considering the vast about of existing data laying around most companies have a good idea about the software will need to handle before coding begins.

Of course, building software around tests is an imbalanced approach. One needs to step back and ask fundamental questions about what the code should be doing ... the case for BDD.

But these questions are already part of the iterative design process.

Anyway, the process of getting my new web site up and running is taking an extremely long time because I am testing every single little piece of code and cranking through some really fundamental questions about the way I want the programs to behave.

Control Languages

I wrote this post in reply to Liz Keogh who said responded to my post on the similarities between "Behavior Driven Design" and "Behaviorism"

"BDD is merely the use of examples in conversation to illustrate the desired behaviour of a system. The approach is all about communication, and nothing to do with behaviorism.

There are some supporting tools which turn examples into automated tests, and this is what Kevin Dishman has criticised. They aren't the only tools, nor do they speak to the heart of BDD.

I think this post indicates some fundamental misunderstandings (and Kevin Dishman's post is misleadingly titled).

December 27, 2015"

The comment made me feel bad; So, I spent the morning reading reading Dan North's introduction to BDD (dannorth.net/introducing-bdd/). In this article, Dan North says he was using an Agile method called Test Driven Design. TTD creates a barrage of tests on the logic of a system.

He was having a hard time wrapping his head around the logic of programs and decided that replacing the word "test" with "behaviour."

"I found the shift from thinking in tests to thinking in behaviour so profound that I started to refer to TDD as BDD, or behaviour- driven development."

Changing this word made Dan North feel empowered. The little switch in Mr. North's mind pushed the logic of the system back a step and gave him the feeling of control.

This is identical to what happened with Behaviorism.

Prior to behaviorism, academics held that man was a rational being. When researchers spoke of human beings, they would talk about the reasons for their actions.

Behaviorists felt uncomfortable in engaging in this discussion of reason and felt empowered when they pushed reasoning aside and decided to focus strictly on human behavior.

Behaviorists felt extremely empowered because they systematically pieced together an ideology that reduced the people around them from rational beings to test subjects ripe for experimentation.

Behaviorism quickly became a favored ideology of the ruling elite and intellectuals. When you reduce people to behaviors, you no longer feel obliged to engage with their reasoning processes. You can treat them as sets of behaviors.

North's works seems to take on a Chomskian flare. Changing a word made him feel better; So he jumped in to create a new language and that made him feel even more empowered.

Dan North is openly admitting that he created BDD as a control language.

I admit, I like to feel in control. So, I would like to ask you a simple question: Is my organizing a department so that I feel empowered really the best course for a company or for society at large?

If I told you that; you would probably look at me and say: No way! I don't want to be subservient to you!

IMHO, development should start with the questions about what is best for the business and what is best for society at large.

Come on, if a business isn't actually improving things for its customers or for society, should it really be in business?

But, lets get back to Dan North.

Mr. North is writing in 2003. If I take his article as if it were the first article ever written on business communications. Then, yes, it sounds wonderful. But it wasn't the first article on communication. People have been yakking on and on about the best way to organize things since antiquity.

The things we have today were built on things from the past.

Dan North was working on, what was then, a relatively new idea called Test Driven Development which was created to overcome perceived faults in Object-Oriented Analysis, which was created to overcome problems of procedural design, which was created to over problems with control structures in basic programming languages which were designed to overcome the vagaries of assembly language which was created to piece together ideas about algorithms and Boolean Logic with computer circuitry. These ideas, in turn were built on ideas dating centuries before.

Needless to say, there are huge volumes of work on system design. One can't just jump in and say: "Oh wow, this idea makes me feel empowered and go it with."

One actually has to ask if this path is taking a better route than the millions of routes tried and followed before.

It is fun to say: "Dan North is just trying to improve communication by creating a new language."

Stepping back, one has to ask: are our communication problems the result of lack of language? There's over a million words in the English language. The language includes tried and true grammatical forms. Why isn't this language good enough?

The language we have is sufficient for communication. The problem is that, even with a great language, it is extremely difficult to get people do what we tell them to do.

Mr. North isn't creating a new language to communicate ideas. The purpose of this new language is to control a process.

BBD falls into a category called a control language.

This brings up a fun story.

Back in 1949, a man named George Orwell was looking at the control languages being created by behaviorists and other ideologues of his day.

The people creating the control languages felt empowered, but Orwell had the foresight to think ahead and tried to imagine what would happen to the people who inherited a world controlled by the new control languages.

Orwell decided to write a piece of speculative fiction about the people who inherited a control language that he called "newspeak" that took place 40 years in the future. He called the book "1984."

The idea of a control language is not new. Orwell wrote about this very subject 70 years ago. I agree with him. Control languages seem empowering at first but entrap the people who become ensnared in the newspaek.

The idead of creating languages to control processes or people is not a new idea.

The issue of the proper use of language has raged since the invention of language itself.

The proper use of language was a prime topic of Socrates through Aristotle. This tradition created an education based on a thing called the "Trivium." The three legs of classical education were Grammar, Logic and Rhetoric.

Grammar studies the structure of language. Logic is about the structure of ideas. Rhetoric is the art of successfully communicating ideas.

The whole point of classical education was to teach students how to develop and communicate their ideas. Dan North is not the first person on the planet to think that language is about communicating ideas.

I find the Trivium to be relevant in programming. For that matter, I would say that the best way to organize software education would be to revive the Trivium.

So, lets look at computer science:

There are many different computer languages that have similar functions. The syntax of these languages maps to grammar. What the computers actually do is called logic. The way we talk about computers is rhetoric.

Interestingly, the heart of Aristotle's logic is a strange thing called a "syllogism." To cut to the chase, an "if/then" statement is a form of the syllogism. Boolean Logic is a subset of classical logic written in symbols for easy manipulation.

We need to be able to talk about computer design. This talking about design maps to rhetoric.

All of the little tricks we pull on our coworkers to win the battle of ideas fall under the guise of rhetoric.

This whole thing we call computer science is very much a natural development from the discourses Aristotle that took place a few millenniums back.

The computer sitting on your desk top is essentially doing what scholastic monks were trying to do in the Middle Ages. Computers are trying to make sense out of an endless stream of syllogisms.

The book "Aristotle's Children" is a fun read showing hows societies blossom when the people embrace logic and how societies falter when power brokers first try to control the use of logic and are later compelled to stamp out reason.

This cycle of reason blossoming and being suppressed by man's innate desire to control others has been repeated time and time again in history.

Orwell's generation was the generation that removed logic from our primary school curriculum as politicians sought the creation of new control based languages. (The left, the right and this thing called ISIS in the Middle East all use control based languages to consolidate power).

So, let's look at the removal of logic from the curriculum.

Anyway, the Trivium formed the heart of education from Aristotle up until the early 1900s.

We had a education which focused primarily on teaching people how to develop and communicate ideas.

The problem is that people who learn to develope and communicate ideas tend to become unruly.

Seriously, people who are skilled at reasoning start to realize that the ideas coming from the king are no better than those from the learned pauper. Even worse, since they can communicate, their discontent with being ruled can spread.

In 1776, there was an event in which classically trained thinkers in the colonies had the temerity to up and rebel against their beloved king! (gasp)

It happened again in France a few years later! (gasp, gasp).

I don't want to get into that spectacle beyond saying that people who learn to reason will question their leaders and become unruly.

After the horrific events of the American and French Revolution, the ruling class retreated into its ivory halls to address the question: How does one reign in all these unruly people?

If language and logic are the source or rebellion, one must create new languages and new logic to reign the people in!

These new creations were called "ideologies." The new ideologies generally feature some form of "modern logic" coupled with some form of control language.

BDD seems to fit this pattern called "ideology." The people who push it seem to fit to a model called "ideologue."

Historically, one of the first new ideologies was created by King William IV and Sir Robert Peel (1778-1850). It is called "conservatism." The century saw a ton of other control languages and logics including socialism, modern liberalism, capitalism, progressivism, etc..

What the ruling class did was truly clever. Both the Left and Right are based on the essentially the same structure. Rule by George Bush and rule by Barrack Obama both result in centralization and a growing state of individual disenfranchisement.

There's not just one ideology based on the modern form. There's hundreds and many of the ideologies re-inforce each other.

Behaviorism is really just one of a whole class of ideologies that feature a control language and new logic. Behaviorism is often applied in political science where political decide on a desired outcome then set in on manipulating the people until they achieve their outcome.

The US Founders dreamed of a society where people, trained in classical logic controlled their government. The removal of logic from the curriculum flipped the switch and we now have a ruling class set on manipulating the people.

After reading North's work, It is clear that BDD shares both a name and structure with this an other ideologies.

As for the need to create a new language, I contend the existing languages work quite well.

Creating a newspeak to reign in software design is likely to make communication harder in the long run as each little fiefdom develops its own newspeak.

I would encourage anyone confronted with BBD at work to resist.

If you really want to learn to communicate, I suggest reading "The Art of Thinking" (aka The Port Royal Logic) written by Antoine Arnauld in 1662 before inventing a new language.

I Don't Need sqlLoop

What was I thinking?

Yesterday I introduced a new function called sqlLoop().

I can do everything in sqlLoop() with repeated calls to sqlRow().

sqlRow() works as follows. If it is successful, it returns an array full of data. On failure, it returns an array in which the first element is false. My loop looks like:

$row =sqlRow($sql);

while ($row[0]) {
  // process row
  $row = sqlRow(); 
}

The one problem with this code is that if I forget to call sqlRow() in the loop, I've created an infinite loop ... and I am very forgetful.

Hey, I have an idea.

Scrambles back to the drawing board.

If you call sqlRow('chk') then the function returns true if the row exists and false if it doesn't. It maintains a counter to that breaks the loop.

$row =sqlRow($sql);

while (sqlRow(DB_CHK)) {
  // process row
  $row = sqlRow(); 
}

The first form is faster, the second protects from infinite loops. I will put this new code up in a minute.

The Bug is with the User and Not the Program

This just in: The program does not have a bug. The problem is the user. Here is the bug report. The response is simply that I need to learn to program.

The exact bug I reported was that, on my system, the PDOStatement::fetch(PDO::FETCH_NUM) command returns an array when called on its own, but it returns an incomplete string when called in a foreach loop. PHP's documentation states the function always returns an array and that foreach loop is the preferred method for looping through output.

The program does not do what their examples and their documentation claims.

This is the classic definition of a bug.

Now that the bug report has been rejected, I feel free to solve the problem my way.

I've worked with dozens of SQL drivers over the years. Long ago I realized that a SQL driver needs to do things. One needs to be able to take data out and put data back in.

Now, the very first thing that I do when I encounter a new database driver is to create my own wrapper for the driver. I usually call the program that sends SQL commands to the database execSQL(). I usually create two programs to pull data out of the database which I call sqlValue(), and sqlRow().

As the name imply, sqlValue() pulls a single value from the database with a command like:

$user_nm = sqlValue('SELECT user_name FROM User_Table WHERE user_id = :user_id);

I never liked the mysql_ or the mysqli_ family of programs because they are vendor specific. So, I simply wrapped the vendor specific programs with my programs.

Now, I am not as pigheaded as you might imagine. I strongly believe in following industry standards. I was ready to abandon my tried and true approach for the new Object-Oriented way.

But I have yet to see anything which is as robust as my little database functions.

So, what I decided to do over Christmas was to give up on the "object oriented" way and revive the tried and true style of accessing data with simple functions.

I did make one huge leap forward in the way that I loop through database tables, which I will get to after discussing loops.

So, I would like you to take a moment and consider the while loop.

The loop has the form:

While condition is true perform an action.

The condition is a statement that evaluates to true or false. If you have a condition that is always false, you will never see the inside of the loop. If a condition is always true, the loop will run until some one or something kills the job.

Now, lets consider the following PHP Code which loops through an array:

$arr = array('a','b','c','d');
$cnt = 0;
while ($val = $arr[$cnt++]) {
  echo 'Line '.$cnt.' is '.$val.'';
}

In the conditional part of the statement. The program attempts to pull an item from the array. If the assignment succeeds, the system reports true and the loop continues. When the assignment fails, the system throws an error and the computer pops out of the loop.

That is a very ugly way to handle a loop.

The foreach loop implements an idea called "traversable." It is functionally the same as the while loop, but captures the error more elegantly.

$arr = array('a','b','c','d');
foreach ($arr as $val) {
  echo 'The value of this object is '.$val.'
';
  if ($cnt++>100) break;
}

In a superficial way, the foreach loop appears more elegant. As my experience of this last moth shows, the condition for breaking out of the loop is dependent a weird thing called "traversable".

The problem in both the first and second loop is that we are trying to use the code that assigns the value as the condition for the loop.

The form of a while loop should be: "while condition perform action." This next example shows a better form for the while loop:

$arr = array('a','b','c','d');
$cnt = 0;
while (isset($arr[$cnt])) {
  $val = $arr[$cnt];
  echo 'Line '.$cnt.' is '.$val.'';
  $cnt++;
}

The function isset() cleanly evaluates to true or false. It returns true for offsets 0 thru 4 and fails on 5; popping us out of the loop.

Looping with sqlRow()

I decided I wanted to improve the looping with sqlRow(); So, I updated the program to work as follows.
If you call the function as sqlRow($sql,$arr), with $sql containing a SQL statement and $arr being an array of parameters. The program will fetch a row with that statement. If you call sqlRow() with no parameters; it returns a row from the last result set. If you call it with the option DB_CHK; the program returns a true/false statement.

A loop works as follows:

$sql = 'SELECT a bunch of data FROM Table WHERE id = ?';
$arr = [$id];  // arr contains a bound array with the variables for the SQL
$row = sqlRow($sql, $arr);  // get the first row.
while (sqlRow(DB_CHK)) {
  // process data in $row
  $row = sqlRow();  // get the next row.
}

The big trick in the code above is that I pass the $row parameter by reference.

Creating the first row outside the loop seems awkward at first. However, it is useful when producing tables, lists and other HTML elements. You want to know if you have any data before creating the table as in the following example:

$sql = 'SELECT list_data FROM Table WHERE id = ?';
$arr = [$id];  // arr contains a bound array with the variables for the SQL
$row = sqlRow($sql, $arr);  // get the first row.
if (sqlRow(DB_CHK)) {
  echo '<ul class="ulclass" >';  // open Ordered List
  $closeUL = '</ul>';
} else {
  $closeUL = '';
}
while (sqlRow(DB_CHK)) {
  // process data in $row
  $row = sqlRow();  // get the next row.
}
echo closeUL;

You don't want to print the UL tags if there is no data.
I put the pre-alpha version of the code in my code viewer. The code has passed my initial tests on this and a few other servers. I will start writing code with sqlLoop() tomorrow.

The Best Way to Connect to the Database

I've been thinking about writing an article on the best way to connect a web server to the database.

I can't write this article because, for some unknown reason, I cannot get PDO to work on PHP7. I've completely removed and reinstalled PHP a half dozen times and it simply doesn't work.

I've spent more time battling PDO than I've spent battling any piece of software in my life.

I've had inconsistent results with PDO on every server that I've encountered. There is something seriously wrong with PDO.

Since I can't get PDO to work with SQLite3. I decided to install PostgreSQL.

I didn't want to install the full application on my account. So, I Googled "postgresql hosting" and found someone reselling Amazon Web Services.

ADVERTISEMENT: You can get access to free (but limited) database hosting with Amazon Web Services. NOTE: This link sets an affiliate cookie. I might get a commission on sales. Of course, you are buying something for $0; So, my commission won't actually be anything, but at least it is a sale. It is better to get a commission on nothing than to not get a commission on anything ... or something like that.

If you don't want to go through my affiliate link, you can just type aws.amazon.com/free/ in your browser.

You can sign up for a PostgreSQL or other database. It will create a and user for you. You can then install the PostgreSQL client on your web server (or home computer) and start accessing the database.

On Ubuntu, you install the postgresql client with the command "sudo apt-get install postgresql-client".

After installing the client, you can access your Amazon hosted database from the command line with psql.

It was like  magic. I was able to create and update tables on the Amazon server.

The next step is to connect PosgreSQL with php. There are different modules for different databases. I installed the module for PHP7.0 with the command "sudo apt-get install  php7.0-pgsql"

And ... it doesn't work. I can query the database, but I am unable to modify the database with the PDOStatement->execute() command.

Anyway, as soon as I figure out how to get PDO to work on PHP 7.0. I am going to write that article on the short comings of PDO.

Unfortunately, I can't write that article until I get the damn thing to work.

Although I am frustrated to no end with PDO, I love having a free database on aws.amazon.com. Here is the affiliate link for those of you who skipped the advertisemnet: Amazon Web Services

SQLite3 Object

I feel like the biggest dolt in the world.

Apparently, PHP has a SQLite3 Class that is similar to PDO. The documentation does not say it works in PHP7, but first tests indicate that it can connect to the database.

I thought PHP had deprecated all of the database functions and wanted universal usage of PDO.  The inability of PDO to fetch SQL queries had effectively turned my new domain into nothing but an expensive Boggle Server.

The SQLite3 Class is significantly different than PDO. For example, you can't just send it an array of data, you actually have to bind variables to make it work right.

The class has different function names and might have a slightly different work flow than PDO.

The class might be sufficient for my needs; so I will start developing with it ... keeping in mind that PHP is likely to deprecate the object in a future release.

=======

Never mind. The SQLite3 Class isn't working either. There are too many goofy things with this object. For example, the query() and execute() functions return different types of objects, the big kicker, you can't use bound variables for queries ... which was the whole purpose of moving from functions to objects.

Boggle Server

I've been staring at an expensive Boggle Server for the last several days and feeling like a dope.

I can't do anything with my web site migration until I figure out the PDO fetch problem. So, I've spent the last several days playing Boggle over SSH trying to figure out how I can get the site running.

Anyway, I decided, after long hours of torment, that I would file a bug report. It bug report 71185. I suspect it will be summarily dismissed, but It will be interesting to watch the rejection process as I master typing short words on my Boggle Server.

SQLite3 Drivers?

I finally figured out the cause of the mysterious PHP problem that has stopped me dead in my tracks.

My problem is that the command PDOStatment::fetch(PDO::FETCH_NUM) is returning a string instead of an array.

Apparently, this is what happened:

The SQL standard (which was established before PHP was a twinkle in Lerdorf's eye) return strings that the receiving program must parse.

The RDMS companies that courted PHP in the early days wrote drivers that presented the data in a more efficient manner.

Big companies, like Microsoft and Oracle, are always happy to provide drivers as the drivers create a dependency on their products.


SQLite is an open source, "zero configuration", standards based program which is designed to be embedded in other applications. The term "zero configuration" means that SQLite does not design drivers. I want to emphasize the term "embedded." SQLite hands you their code and you actually embed it in your program; so there is no driver.

Anyway, the problem I am experiencing today appears to be a repeat of a bug reported experienced in 2008.

If you read the report, the PHP programmers say that this problem is the fault of the drivers between PHP and the database. (The drivers take the standards based output and put it in a form more to the liking of PHP).

SQLite is standards based and embedded. SQLite has no drivers. In the SQLite paradigm, SQLite presents source code to the world. Companies like PHP can compile it into their program ... the catch is that it is the responsibility of the company embedding SQLite to make the driver work.

This error is actually quite humorous. This grand PHP PDO object (which is supposed to be the crown jewel of the new PHP) can't handle standards based output.

PDO has a dependency injection. The PDO object is dependent on the external driver written by large firms like Oracle and Microsoft.


I know that it will be auto rejected, but, I am seriously considering writing up a bug report on this issue, but I think I will try a twitter poll first.

Repeating Mistakes from Philosophy into Code

There is an interesting trend. The academic community takes popular ideas from philosophy and tries to impose them on computer engineering.

Behaviorism is a weird ideology that appeared last century that de-emphasized individual reason in the study of human behavior in favor of the idea that man is simply driven by impulse.

Historically, Behaviorism played a role in the decision to remove logic from the curriculum last century.

Apparently there is a new buzzword in soft ware design called BDD (Behavior-Driven-Development)  that is becoming popular in agile design circles. I've been reading buzzword rich articles trying to sell this technology.

It appears that the goal of BDD is to create a group-think approach to software development that creates something called "Domain Specific Languages" that spews out code.


Kevin Dishman wrote a blog post called BDDont questioning BDD as a design tool.

I want to applaud Mr. Dishman for standing up and calling out this technology.

The ideology of Behaviorism has not improve society.

Ideas from behaviorism have influenced both the left and right, and we see that many politicians and people are making their decisions based on impulse and are not reasoning through ideas.

How can we reason through ideas if we never learned logic?

What we have now is a political system with media feeding us an unending stream of images which cause impulses which then cause us to vote and tweet in predictable ways.

The factions in our political system use Domain Specific Languages. The result of removing the study of logic from schools and emphasizing behaviorism is that the people at large have become tied up in factions and cannot communicate with each other.

Personally, if I was working for a company that tried to impose BDD, I would probably just leave. But, you know, the imposition of BDD in the work force creates a situation where one can actually measure the results of development driven by a group-think process to ones in which individuals engage in reason.

The latter, reason, produces better results than group think. I hope Mr. Dishman keeps up his scrutiny.

After Hours of Debugging I Discovered This

Two days wasted. I've been running the following code on PHP verion "7.0.0-6+deb.sury.org~trusty+1"

  $sql = 'SELECT id, test_nm FROM Test_File LIMIT 3';

  $db = new PDO('sqlite:/var/www/db/main.db');
  $stmt = $db->prepare($sql);
  $stmt->execute();
  $row = array(1,2,3);
  echo gettype($row).'</ br>';
  // This reports row is an array.

  foreach ($stmt->fetch(PDO::FETCH_NUM) as $row) {
    echo 'This should be an array it is a '.gettype($row).'.
';
    // this reports that row is a string.
  }

This is the result of the query on my server: https://yintercept.com/resources/new.php.
The PDO::fetch() directive on this version of PHP returns a string. The description on PHP.NET says: "PDO::FETCH_NUM: returns an array indexed by column number as returned in your result set, starting at column 0"

I got my compiled version of PHP from lauchpad.net. I guess I am stuck waiting for Ondřej Surý to update this package to PHP 7.0.1 to figure out if this bug is related to something I did or this version.
This failure brings me back the the question: Are objects superior to procedures? Here is an object a production version of PPP 7 causing total failure in the system I am trying to port to PHP 7.

This particular failure took several additional hours to debug because I assumed that the problem was something to do with the path I took through the object. It didn't even dawn on me to actually check the type of the data coming out of PDO::fetch() as the manual said it was an array.

NOTE: A 2008 bug report says the problem is with the database driver and not PHP:

https://bugs.php.net/bug.php?id=44341