Tuesday, May 3, 2016

Maltego 4 - it's finally time...


Maltego 4 is finally ready...click on the picture below to view the release video:

Download the software [here]

...but if you want to know more...

The Maltego 4 story

In March of 2015 myself, Chris and Andrew sat in a room in Cape Town to decide which feature to build next. It's one of the hardest challenges managing Maltego - deciding what to do next. There's always at least five major features competing for our attention. Be that geospatial view, temporal view, feeders or a browser plugin - there's always the next big thing waiting. We argued the entire day, everyone having their own favorite. At around 7 o clock we were tired, hungry and irate. I asked Paul (at the time still pretty green and struggling to keep up with all the intricacies of a new design) "if you could have any feature in Maltego - what would it be?". He didn't have to think long and answered "handling big graphs". Then he casually put his headphones back on and ignored us.

It wasn't what I wanted to hear. We kept on ignoring the issue to the point that we almost believed it wasn't a problem anymore. We didn't want to fix it. It was hard to fix. It meant ripping the guts out of our product. We all knew that it would mean many months of nothing but rebuilding things we already had. No new features, no new flashy bits. Just hard work - rebuilding Maltego from the ground up. But Paul was right. It wasn't the popular answer, but probably the right answer.

For the months to follow we had no new features coming out. We issued a couple of patches for Maltego Chlorine (3.6) and kept supporting the old version. I asked Chris and Sonja if they had a rough idea on when we'll be done. The first date we tried for was Black Hat Las Vegas 2015. August. By June we all knew it was way too early and we pushed it back to Christmas 2015. In early December 2015 they sent me a barely working version. It included lots of disclaimers on which parts I could play with - but it could handle 30 000 nodes with ease. It was exciting, so exciting that I had to make a video about it. We decided we needed a new website too. Paul was to run with that - it had to be ready to go with the release of the new version.

Putting back all the pieces took longer than we anticipated and we hadn't even started on collection nodes - the secret weapon in the fight against large graphs. Collection nodes were not a new concept. We tried it back in 2009 and never released it - it failed miserably, partly because the product (and perhaps we ourselves) were simply not mature enough. The trick then was usability and the usability of collection nodes was a major struggle now. We decided to completely redo the interface. The version I had in my hands looked really bad. The user experience was bad. It was riddled with bugs, things that simply didn't work. I pulled the video. It said we'll have it before 2016. There was simply no way we'd have it done. Christmas came and went and we had nothing.

During January 2016 I felt like the new version was never going to happen and that, even if we did get it right, users would hate it.  I didn't even want Andrew and Paul to try this version because it would leave a bad taste in their mouths. But we kept slogging and gradually things started to get better.

The turning point was early March 2016. After many usability / look /feel meetings we were slowly getting there. Things started to fall into place. It was looking the part and after several iterations the interface was starting to behave the way you expected it to. Preparing for a conference in April I exclusively used the new version. Using it in anger for the first time it was clear that this was something really special. All of the hard work was starting to pay off. Things that only lived in our imagination for a year were now right there on the interface, and it was working exactly the way we envisioned it. It was fast - terribly fast. And slick. And it handled almost anything I could throw at it. There would be no going back to Chlorine ever. It was time to set a date for the final release.

The date was set to be the first of May 2016. But that was a Sunday so we went with May 2. This was a public holiday in South Africa (and in many other countries) so we went with Tuesday May 3. Now we had to tie up all loose ends (memory leaks, branding, testing/fixing/testing/fixing). We contemplated calling the new version Plutonium, but this release was so different to anything we've had in the past that we decided it would be easier to just go with 'Maltego 4'. We sent out betas to a select group of trusted users. The feedback was phenomenal. They loved it.

We made a 'camera-ready' release  on the 26 of April and I flew to Cape Town to go make the release video. We shot an afternoon, an evening and the next morning and I flew back to Gauteng to edit. After some hiccups the final edit was ready on the Sunday before the release.

Today is Monday. Tomorrow we release. A brand new website, a brand new product. The release is not perfect. There are always things we want to improve and there are most likely a few minor bugs that we'll squash over time. With a system as complex as Maltego it's almost impossible to achieve perfection and I have to constantly remind myself that nobody cares about Maltego as much as we do. It's a child we all raised together as parents,siblings and a crazy uncle.

Some other stuff we probably need to say

Maltego 4 comes in two commercial flavors. Classic (the standard version) and XL (the pro version). The *only* difference between the two is that Classic is capped at 10 000 nodes. Oh wait - and the price - Classic is still $760 and XL is $1800. We had lots and lots of discussions about the price. We haven't raised the price on Maltego for a long time and we didn't want to raise the price for the new version. So we decided to split it into two products (we've been wanting to do this for a while now). We then had to decide what's in the XL version and what's not. An easy out would have been to exclude collection nodes from the Classic version. But collection nodes are super useful - even when working with small graphs as they quickly show you where you need to (probably) look - NOT at the collections. So collection nodes stayed. Then it was crippling Classic in some way...but that just felt wrong and so we didn't. Every time we thought about taking things out of Classic we cringed. Finally we decided capping the total number of nodes in a graph. But where to cap it? We decided on 10K nodes because of two reasons - the first being that in the past, working with 10K nodes would be painfully slow - so - we weren't taking functionality away from anyone...as they never had it. Secondly the slider was always maxed out of 10k - it didn't make sense to have it at a lower number. 10K it was.

Still more stuff

Maltego Chlorine users will be able to simply download Maltego 4 Classic and activate it with their license key. No upgrade fee to Maltego 4. Users that wish to upgrade to XL should just pop us an email.

Then there's the question of the community edition. Ye - we're no longer supporting it and we'll be removing it from our site. Hehe.. no. Give it a bit of time. We'll create Maltego 4 CE and Maltego 4 Kali soon. No really. We will. Currently the CE versions are still using the old tech.

And finally..

One last thing. CaseFile. The one we always leave behind at the bus stop. There's good news. With Maltego 4 being so totally amazing we're making CaseFile completely free. No registration. No nothing. Just download and use. And in time we'll upgrade CaseFile to goodness of collection nodes, large graphs and a face lift.

Right, that's about it. We're super excited to see what you think about all our new tech. It's been a long journey and we're really pleased with our progress. We hope you're too!

RT and the rest of the (tired) team.

No comments:

Post a Comment