Monthly Archives: November 2014

SMOSLT Development Status

SMOSLT is free software (free as in free beer) and is supported by a single person.

Development of this software might stop at any minute, should it’s author become otherwise occupied, such as involvement in contracts.

For the moment, it is in active development.

SMOSLT.options Module

The SMOSLT options feature is what runs your assumptions against your schedule with the options that you choose to compare with each other.

 

The Options Module

In the codebase it goes by the name of smoslt.options

 

For the rest of this document, we will refer to this module as simply “options” or “options feature”.

 

The options module, like all other modules in the SMOSLT system, is an OSGi bundle or jar, and has almost all other SMOSLT jars as dependencies. It also has optaplanner as a dependency.

 

Everything else in the SMOSLT codebase is deterministic by comparison. The options feature allows you to compare options that are not even realistic to compare in a more deterministic way. The options feature runs on the optaPlanner backbone, which allows you to compare combinations of options which are NP-Complete by their very nature.

 

NP-Complete mathematics is too big a topic to discuss here, but that’s OK – because it’s amply covered in wikipedia here.

 

How Does It Work?

 

The options feature is designed to be almost completely decoupled from the rest of the SMOSLT codebase.

 

The options module maintains no awareness of SMOSLT.stacker or how it does it’s job. The options module thinks of it’s options as an ordered list of option names. It’s job is to simply toggle on or off different combinations of options, and then hand back a list of options that are toggled on to get a score – in this case, from SMOSLT.stacker.

 

Example:

 

options is given these 5 options, in this order.

  • option1
  • option2
  • option3
  • option4
  • option5

 

First, it might toggle only option2 and option5 on, and then tabulate the score for that. To do this, it hands off a String “option2 option5” to SMOSLT.stacker, which returns a score of “-217, -486, -344”

 

So now the options database has this information stored in it:

“option2 option5” = “-217, -486, -344”

 

The options module continues this process, gathering scores persisting them, for every combination of 1 to 5 options that the options module is interested in. When it is time to terminate the run, the options module has gathered a handy little database of many different combinations of options, and a score for each combination.

 

The options module neither understands, nor cares about, how it’s scores are arrived at. But it does care about the scores themselves. More about that below.

 

Why Does SMOSLT.options Module Care About Score?

 

Since the options module does NOT care about HOW the score was achieved, why does it care about the score itself?

 

Imagine that the example above was more typical of a real software project, and that instead of 5 options, the real list was more like 100 or more options. Calculating every combination of this many options might be more work than you would wish to throw computing power at.

 

For optaPlanner to do it’s job, it needs to use the score to evaluate which combination of options to try next. If, for example, option3 and option5 in combination has so far always yielded an unworkable score, then it might decide not to keep trying those two in combination with other options, or at least to not try that combination until other more promising combinations of options were scored first.

What Else Does SMOSLT.options Database Persist, and Why?

Shown in the example above, every combination and score would be persisted for you to do analytics with, using another tool. So it might look like this:

“option2 option5” = “-217, -486, -344”

“option2 option3 option5” = “-313, -486, -344”

“option1 option4” = “-682, -108, -751”

 

This would be sufficient for you to take into the SMOSLT.analytics piece and summarize your options in charts and graphs, but once you used the various charts and graphs to analyze your best options, you still might wish to know the actual schedule that SMOSLT.stacker used to calculate your score.

 

For that reason, a binary of the schedule is also persisted with each score. So the real persistence table looks likes this:

 

“option2 option5” = “-217, -486, -344”, [binary]

“option2 option3 option5” = “-313, -486, -344”, [binary]

“option1 option4” = “-682, -108, -751”, [binary]

 

What Do The Various Score Numbers Mean?

The first [n] numbers are pass-fail. Only zero is a pass. Everything below zero is a fail.

The rest of the numbers are soft criteria. The only thing that the options module cares about is their size (the closer to zero, the better).

To the options modules, these are just numbers. To the stacker and analytics modules, these represent real concerns.

Related Information

  • smoslt.stacker module
  • smoslt.analytics module

SMOSLT Project/Modules Summary

You probably won’t understand SMOSLT as an app until you first read why SMOSLT is run within an IDE. It just won’t make sense.

SMOSLT.app

This is the main SMOSLT project, where all the important action happens.

 

These actions include

  • running the other modules
  • Taking the assumptions you give it, and coming up with a list of costs based on those assumptions.

SMOSTLT sequence of events within modules:

 

Orchestrated primarily from app module, but also manually from within IDE, given no UI

  • before firing, user has already
    • created a compliant PL file
    • created list of options and other assumptions in assume module
    • created SideEffect code that drives each option.
  • imports PL file
  • imports assumptions from assume module
  • sends each of these to the optaplanner options module
    • options
    • schedule binary
    • analytics
  • optaplanner options module then folows this sequence
    • toggles one option on or off at a time
    • sends that combination of options to stacker
      • which can be on separate threads or machines if required
      • which writes each score and binary to analytics module
      • which also then returns score back to options module
    • runs to some reasonable termination whatever that means
      • brute force if options list small enough
      • more elaborate search process if options list too big
  • user then reviews run in analytics module

 

SMOSLT.domain

Java classes consumed by other modules. Kept separate just for purposes of clarity.

SMOSLT.given

Assumptions are required for SMOSLT to do anything meaningful.

 

This is YOUR area, because you are responsible for all assumptions, even though you might start out with 100% default or partially customized assumptions provided by others. This is like what the cop tells you: “Ignorance is no excuse” the results you get will be no more satisfactory or correct than the assumptions you used to initiate a specific run.

 

Areas that assumptions cover:

  • Unit costs for include.
  • Beliefs about if-then consequences – if I have a java task with no testing, then I will get 30% more unanticipated work fixing bugs. That’s a belief. No one can know what really happens until it does.
  • Story templates. If every task is a story, per agile approaches, then the sum total of all tasks is a narrative arc that makes up a story template. No story template is truly representative of what really happens, but to anticipate costs you have to start somewhere.
  • Actual ProjectLibre files. Ease of use demands that you start with a ProjectLibre template, compliant to SMOSLT specifications.

SMOSLT.stacker

Stacker, or ScheduleStacker, creates a real schedule from a ProjectLibre specification template. Stacker knows nothing about SMOSLT or all the fancy stuff that SMOSLT does, it just stacks up tasks and allocates resources like it is told to do.

 

Stacker is designed to be as simple and fast as possible, because it might get called to re-stack a schedule thousands of different times in a single session.

 

Stacker could theoretically be used by anyone, without any SMOSLT usage. It is not anticipated that anyone would wish to do this, but it is welcomed if desired.

SMOSLT.mprtxprt

SMOSLT does it’s active work in SMOSLT code, and does not interact directly with ProjectLibre APIs. A ProjectLibre schedule is entirely converted into SMOSLT code, and then when done entirely converted back into ProjectLibre file. Ne’r does the twain meet.  This conversion happens in this module.

 

Only the brave should ever crack open this module. ProjectLibre APIs can be quite challenging to the uninitiated, and a giant black hole of time. As of Oct 2014, it is being rewritten ground up anyway.

SMOSLT.main

Someday SMOSLT may be runnable from a real GUI, like you would expect from any decent application.

That day has not yet come. So until then, it is run from a command line. This command line , or CLI code is maintained here.

 

SMOSLT.options

See separate document

 

SMOSLT.analytics

 

 

Is This Your Story? Joe the Architect

Imagine Joe Architect as some guy architecting a large software project/team. Also imagine Thoughtworks as a representation of the latest thinking about how Joe might approach his job with the greatest effectiveness. The Thoughtworks reference is, in this case coincidental, it’s could be any set of disciplines that apply to Joe’s work.

 

The primary constraint is that Joe has a reasonably limited budget, he works for XYZ corp which is properly funded, but still not working in a cost vacuum. Joe has both time and budget constraints that prevent him from going absolutely hog wild. He has to deliver something within some kind of limits. He can’t simply hire Thoughtworks, much as he would like to.

 

Joe is a good student. He reads and absorbs all of Martin Fowler’s stuff and also sat through Neal Ford’s 7.5 hours on Continuous Delivery (Jez Humble yada) at the last NoFluff. This is all still very high level for Joe – even though he is a good student – as he knows that any single slide in the presentations he has watched and absorbed might represent days or even weeks of real life implementations to get it all in place.

 

The problem space is similar to the “Pick Two” sign on the wall of his car mechanic’s shop. “You want it fast, high quality, and cheap? Pick any two” Joe has literally hundreds of options available to his team, from how to refactor his code, to which of dozens of persistence stores to use, to which continuous delivery approaches to implement. He evaluate it all, and he has to direct his team without complete information. Only it isn’t pick two of three, it’s pick 5 of 300 great options.

 

Worst yet, the problem space is NP-Complete (see http://en.wikipedia.org/wiki/NP-complete) – even if you could give the problem to a computer to solve, there would be so many options that the computer could grind for days without producing it’s first decision. Just too many combinations of options to consider using brute force alone.

 

You know what Joe does already. He just holds a wet finger to the air and makes some decisions and his team goes to work.  That is the problem space. Does Joe have any better combination of options than just going with the direction that the wind is blowing?

 

Joe has too many great options.

 

Here’s the real kicker:

It isn’t just that Joe has a problem that he can’t solve effectively.

 

The real loss is that there may be one or more options, ones that he doesn’t know about, that completely transform his job from a moderate success or even partial failure, to a runaway kick ass success! Options that really make a difference. But how would he know?


He might entirely miss the important options. Lost opportunity.

SMOSLT vs Other PPM Software

PPM is a big category

SMOSLT does not attempt to compete with other PPM software. There is plenty to compete with. Use what works best for you.

SMOSLT can be compared with other PPM tools

SMOSLT’s place in the mix is to be fast and relevant to both developers and project managers.

SMOSLT is intended to be ridiculously flexible and adaptable. No good software can do that. That’s why SMOSLT is, happily, such crappy software.

A fantastic list of other PPM tools is compiled here

http://www.prioritysystem.com/tools.html

SMOSLT – Runs in IDE Only

SMOSLT could have been written as great, standalone app. Even now, that could still happen. But it won’t. Why?

SMOSLT is intended as crappy software

My name is Pete Carapetyan. I know how to write great software. SMOSLT is not written as great software.

Instead, SMOSLT is barely held together with chewing gum and bailing wire. Happy Path Software at it’s worst.

This is by design.

SMOSLT is barely usable

… and so it shall remain. To use SMOSLT you probably have to be a java developer, and you need a modest amount of patience.

You probably have to follow the youtube just to figure it out.

You can probably figure out 1000 ways that SMOSLT can and should be improved.

Why not improve it? You could make a great product!

Indeed. There are probably much better ways to spend one’s time. SMOSLT is about making other software great, not about making itself great.

Gotta pick your battles. Plenty of other battles to fight.

Improve your own software, not SMOSLT

SMOSLT is just a means to an end. Don’t improve it. Instead, spend that time improving your own software.

Unless of course you can’t help yourself. If that is the case, send your patches to pete@datafundamentals.com

But my own stuff is being mis-represented! That must be fixed!

Ahah! Different topic of conversation! Your own [insert technology here] is being mis-represented? How can I fix that?

Is there an easier way?

We could probably ship SMOSLT as a fully self contained virtual machine. IDE, projects, etc. Just launch and run.

No, we haven’t done that yet.

 

SMOSLT:generator – A Project Generator

This project is front ended in SMOSLT.webgen and is only now partially complete


 

The Project Generator generates up to an infinite number of [allegedly] representative software development projects of varying types, sizes, and base option levels.

 

Generated projects can be used by SMOSLT to operate on, giving it a wide array of situations to test various options and hypothesis against.

 

Representative Project Types

The following project types are provided for by the generator:

  1. Business API
  2. Mobile App
  3. Content Delivery
  4. Batch Transform
  5. Big Data ETL & Analysis
  6. Desktop Data Tool
  7. Corporate BI Rollout
  8. Streaming Data Input Service
  9. ETL and Analysis
  10. something else

 

Each of these project types is outlined below in Project Type Details. Each of these is also based on a Master Project Template which provides common tasks that all projects might need such as a project requirements task.

 

The Easiest Way To Consume SMOSLT Runs?

SMOSLT can be time consuming and detailed to set up, and there is very little or no benefit for getting it perfect.

 

This low payback for perfection is easy to understand. SMOSLT is for evaluating a wide array of options at the crude and approximate level, not for microscopic decisions based on having to get it right before you start. Just as you wouldn’t use SMOSLT to design perfect NASA space launch software, neither would you try to perfect NASA space launch project design without first deciding exactly what database, network, and operating system options you were going to deploy the new software on.

 

Instead, you might wish to start with a surrogate schedule provided by SMOSLT generated projects. Start with the closest surrogate to your own project, then either use it instead, or use it as a starter for your own schedule.

 

SMOSLT Generated Project Schedules Are Still Not Complete Finished Schedules

A project schedule is never flushed out until AFTER the final SMOSLT options run. This can be confusing. To make it even more confusing, here is the sequence of project schedules as created in SMOSLT. All are designed to be automated as much as possible, since the main objective is not a real schedule, but to know how the options would play out in this schedule.

 

  • Master Template Schedule
  • [Project Type] Template Schedule
  • [Your] Project Schedule Before Options
  • [Your] Project Schedule After Selected Options

 

A project starts off by being copied and pasted from a Master Template, that gives you the basic structure and common elements.

 

This is then given tasks specific to a Project Type. You may now run this through the SMOSLT.stacker, but you probably want to take a couple more steps first…

 

If you copypaste this generic [Project Type] Template Schedule and customize it to be more representative of your own project, such as increasing or decreasing team size, number of repeat tasks, it is nos [Your] Project Schedule

 

Now, apply the options you wish to consider, and you have a meaningful data dump that may be representative of the kind of options available to you

Task Names as 4 Field Array

 

Generated projects start with a template created in ProjectLibre.

 

 

Project Type Details

Master Template

Master Template is an almost empty ProjectLibre schedule with 3 tasks borrowed quite artificially from Disciplined Agile Delivery. These tasks are not regarded as representative of most organizations, nor is this an endorsement of that approach. Something had to be used as a placeholder.

For more on the Disciplined Agile Delivery, see

 

 

 

Please add any tasks which are common to your organization’s

Business API

Business API Project

Exposes a wide array of business data to a common persistence store, exposing it as an external REST API

Mobile App Project

Allows customers to explore current special in-store sales for each store location, to encourage customer visits.

Content Delivery Project

Maintains server updates and pumps latest content releases to each of website, facebook, linked-in, twitter, and instagram.

Executive Dashboard Project

Integrates data from multiple sources into a data warehouse and builds an executive dashboard.

Big Data ETL & Analysis Project

Takes in a wide array of business data into an infinitely expandable persistence store, runs transforms as required, and performs analytics to maximize product pricing opportunities.

Desktop Data Tool Project

Manufacturing line management tool brings in information from production line sensors and databases to graphing and tabular reporting UI.

Corporate BI Rollout Project

Rollout of BI tool to mid level production managers for access to real time and historical production and sales data.

Streaming Data Input Service Project

Integration project bringing in both real time and historical production data, presenting as a single real time data set.

Batch Transform Project

Uses external computing resources to transform massive quantities of website and sales traffic data into usable structured files for other downstream data consumers.

SMOSLT code setup

SMOSLT is an application that is designed to only work from within an IDE, by someone who is modestly capable as a Java developer.

If you need more rationale on why this is the case, see this page.

How to set up a SMOSLT in Eclipse IDE

assumes an eclipse Luna IDE with current 1.7 JRE

 

put this in a shell script and run it:

git clone git@bitbucket.org:datafundamentals/smoslt.analytics.git

git clone git@bitbucket.org:datafundamentals/smoslt.app.git

git clone git@bitbucket.org:datafundamentals/smoslt.domain.git

git clone git@bitbucket.org:datafundamentals/smoslt.given.git

git clone git@bitbucket.org:datafundamentals/smoslt.main.git

git clone git@bitbucket.org:datafundamentals/smoslt.mprtxprt.git

git clone git@bitbucket.org:datafundamentals/smoslt.options.git

git clone git@bitbucket.org:datafundamentals/smoslt.optionsapi.git

git clone git@bitbucket.org:datafundamentals/smoslt.poms.git
git clone git@bitbucket.org:datafundamentals/smoslt.stacker.git

git clone git@bitbucket.org:datafundamentals/smoslt.util.git

git clone git@bitbucket.org:datafundamentals/smoslt.workspace_root.git

git clone git@bitbucket.org:betterology/org.btrg.uti.git

 

checklist

open up luna in [smoslt] workspace

import all projects

set up poms project as maven

run poms project in maven clean install skip tests

run org.btrg.uti in maven …

run workspace_root maven clean install skip tests

[all of the git connections say petecarapetyan in poms that is broken]

SMOSLT.given Module

The SMOSLT given module holds the information that is “given”, as used in the following sentence: “Given these assumptions and this ProjectLibre project file, the SMOSLT app runs the various options and stacks up a schedule of tasks”

 

Parts:

  • SideEffects (Options) – written in Java

 

How SideEffect Classes Are Written

  • Must be placed in the smoslt.given.sideeffect package
  • Must implement the SideEffect interface
  • Operates usually by having one of these effects
    • modify task duration
    • create a new task of a new duration
  • Typically filters task based on
    • Name of task starts with ….

 

see separate document on Saturation vs MoreIsBetter scoring options

 

Impossible to Know What is Possible

The mathematics of creating software has turned upside down.

There is a sweet spot, some mix of consuming work by others, and just simply re-creating it. Finding this sweet spot is getting to be a ridiculous proposition. Not because the options are too few, but because they are too great.

Pick the right set of building blocks, and you can develop anything super quickly:

  • Platforms
  • Tools
  • Languages
  • Practices
  • Sequences

This has become a ridiculously time consuming process. Just knowing what options to pick from can consume an entire career. Picking the right combination of hundreds, even thousands of options? Mathematically it’s absurd to even think about it.

Obviously there are many approaches to solving this problem:

  • The glaze over approach – study options until you glaze over. Then pick what you studied.
  • The social media approach – ask your friends, follow social media.
  • The FoxNews approach – just know what is right. If you disagree, you are wrong. There, done.
  • The career halt approach – just stop being productive until you’ve studied every option
  • The management by following approach – pick the latest offering by big, credible vendors like Oracle, IBM

SMOSLT is an attempt to add one more option to this list. It is a variation on the glaze over approach. It doesn’t solve the problem, it just delays the moment of glaze over, letting you consider options a few more minutes longer before glazing over, and hopefully without halting your software development career in the process.

  • The SMOSLT approach – present combinations of options in terms of palatable math, metrics.

SMOSLT won’t feed your dog automatically, solve world hunger, or get your favorite politician elected. It may help you take a few more deep breaths before charging off and building software with ill considered technology combinations that end up sucking the life force out of your body. If it accomplishes even a part of that modest goal, it’s a big win.