Friday, June 28, 2013

Build a Cloud Paris recap

On June 19th we had a Build a Cloud Day in Paris, 60 of us gathered to learn about Apache CloudStack and ear from CloudStack users, integrators and ecosystem partners. We had four great sponsors that helped make the event possible: iKoula a public cloud provider in Paris, Usharesoft a software provider, Apalia a CloudStack integrator and editor of Amysta a CloudStack usage and metering plugin and OW2 an open source consortium. I finally got all the slides and I embed them in this post for your enjoyment. The day started with an intro about BACD and a presentation of the various talks that we were going to have. I talked about the Apache Software Foundation (ASF) and showed how several Apache projects could be used to build a complete cloud infrastructure.

Ikoula followed with a presentation on how to use CloudStack to build a public cloud. Public clouds are the natural evolution of traditional hosting providers, ikoula offers several cloud services based on a default CloudStack install. With 1,000 servers and approximately 8,000 Virtual machines in the cloud it is a perfect example of a public cloud in production with CloudStack. In the afternoon, Joaqium Dos Santos gave an introductory demo on the CloudStack API and CloudMonkey.

Following a presentation on public cloud, Florent Paillot from INRIA talked about the national continuous integration platform. A built and test system for INRIA researchers. Private cloud used for build and test are a common use case for CloudStack. Florent shared the design and reasoning that went into choosing CloudStack. He also talked about operational details and some issues that he will bring back up to the CloudStack mailing list.

With Public and Private cloud use cases behind us it was time to talk about the software that are available to uses to bring added value and ease of use to their cloud. The first presentation was from UshareSoft, a software that makes image creation and management a breeze. UshareSoft also offers a marketplace ala App Store and a migration service. In the afternoon we had a complete hands-on demo of USharesoft, which boasts a sleek user interface and intuitive image management capabilities.

Activeeon followed just before lunch. Activeeon had already presented at the BACD in Ghent back in January, Brian Amedro the CTO talked about ProActive and its CloudStack Plugin. Proactive is used to manage complex computational workflows in industry such as the financials, pharmaceuticals and video editing. The CloudStack use case was particularly telling: working with a customer they used the CloudStack plugin for video rendering on-demand.

In the fall I had the opportunity to work with a group of students from the Ecole des Mines de Nantes. They developed a CloudStack plugin for BtrCloud, a software that can optimize the placement of virtual machines in the data center. BtrCloud presented the latest development of this plugin, showcasing an embedded user interface and the ability to create various rules (e.g affinity, anti-affinity, spread, pack..) for migration. BtrCloud promises to be very interesting as it could be used for Green IT in the Cloud.

We ended the day with a presentation on cloud usage metering with Amysta. Cloud usage is an internal functionality of CloudStack, which keeps track of all usage in a MySQL database. Amysta extends the core CloudStack metering capability with an embedded interface to create very fine grained usage reports on all resources. Pricing and currency can be set and reports generated in various formats for organizations or individual users.

And that was it in Paris, we will certainly plan another event as everyone was very interested to learn even more. iKoula offered to host a regular CloudStack meet-up, stay tuned for information on that front !

Wednesday, June 26, 2013

CloudStack support in Apache Libcloud

A vote has started on the libcloud dev list for the 0.13 release. The release notes detail all the new features and fixes. I am stoked about this because a few patches that I submitted are included in this release candidate. I patched the CloudStack driver quite a bit to improve the support for Clouds with a basic zone like Exoscale. There is more work to do on this driver including better support for Advanced zone especially for port forwarding, firewall rules and more unit tests. A the CloudStack hackathon last sunday @pst418 submitted a few patches for unit tests and they made it as well into 0.13 RC, terrific.

If you don't know libcloud, it's a python based API wrapper to abstract the various cloud APIs. With libcloud you can create connections to multiple clouds, potentially using different API. At a high level, it is similar to jclouds for JAVA or deltacloud written in ruby. There was already a CloudStack driver but its functionality was limited. If you grab my quickie libcloud shell, you can try to follow this walkthrough of what you can do with libcloud and a CloudStack basic zone. Of course you will need a CloudStack endpoint.

Start the libshell and check what zone you are on:

$ python ./libshell.py 
Hello LibCloud Shell !!
You are running at: https://api.exoscale.ch/compute
>>> conn.list_locations()
[<NodeLocation: id=1128bd56-b4d9-4ac6-a7b9-c715b187ce11, name=CH-GV2, country=AU, driver=CloudStack>]

You might notice a wrong country code, it is hard-coded in libcloud, I need to file a bug for this. Get the list of templates (or images in libcloud speak):

>>> conn.list_images()
[<NodeImage: id=01df77c3-0150-412a-a580-413a50924a18, name=Windows Server 2008 R2 SP1, driver=CloudStack  ...>,
<NodeImage: id=89ee852c-a5f5-4ab9-a311-89f39d133e88, name=Windows Server 2008 R2 SP1, driver=CloudStack ...>,
<NodeImage: id=ccd142ec-83e3-4108-b1c5-8e1fdb353ff9, name=Windows Server 2008 R2 SP1, driver=CloudStack ...>,
<NodeImage: id=f2101a0c-eaf7-4760-a143-0a5c940fd864, name=Windows Server 2008 R2 SP1, driver=CloudStack ...>,
<NodeImage: id=77d32782-6866-43d4-9524-6fe346594d09, name=CentOS 5.5(64-bit) no GUI (KVM), driver=CloudStack ...>,
<NodeImage: id=29cba09f-4569-4bb3-95e7-71f833876e3e, name=Windows Server 2012, driver=CloudStack ...>,
<NodeImage: id=ee241b47-4303-40c8-af58-42ed6bf09f8c, name=Windows Server 2012, driver=CloudStack ...>,
<NodeImage: id=754ea486-d649-49e5-a70e-5e5d458f0df0, name=Windows Server 2012, driver=CloudStack ...>,
<NodeImage: id=0f9f4f49-afc2-4139-b26b-b05a9f51ea74, name=Windows Server 2012, driver=CloudStack ...>,
<NodeImage: id=954752a8-0486-46bb-8e3f-0adb3e01c619, name=Linux CentOS 6.4 64-bit, driver=CloudStack ...<]

I cut the previous output, but there are also Ubuntu and CentOS images on that cloud...You can then list the various instance types or sizes in libcloud speak.

>>> conn.list_sizes()
[<NodeSize: id=71004023-bb72-4a97-b1e9-bc66dfce9470, name=Micro, ram=512 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=b6cd1ff5-3a2f-4e9d-a4d1-8988c1191fe8, name=Tiny, ram=1024 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=21624abb-764e-4def-81d7-9fc54b5957fb, name=Small, ram=2048 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=b6e9d1e8-89fc-4db3-aaa4-9b4c5b1d0844, name=Medium, ram=4096 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=c6f99499-7f59-4138-9427-a09db13af2bc, name=Large, ram=8182 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=350dc5ea-fe6d-42ba-b6c0-efb8b75617ad, name=Extra-large, ram=16384 disk=0 bandwidth=0 price=0 driver=CloudStack ...>,
<NodeSize: id=a216b0d1-370f-4e21-a0eb-3dfc6302b564, name=Huge, ram=32184 disk=0 bandwidth=0 price=0 driver=CloudStack ...>]

What I added is the management of ssh key pairs and security groups, you can now list, create and delete both keypairs and security groups, and use those when deploying nodes. (Don't bother trying to do anything with the keys below, I deleted everything.)

>>> conn.ex_list_keypairs() 
[{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}]

>>> conn.ex_list_security_groups()
[{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'Default 
Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr': u'0.0.0.0/0', u'protocol':
u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'0.0.0.0/0', u'protocol':
u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name':
u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test',
u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}] >>> conn.ex_create_keypair('test') {u'privateKey': u'-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQCAuPufJnbzFkyIUaMymkXwfilA5qS2J9fx2Q3EWNDs1m89FLFq\n19ERjG43ZvRuI/KGwWZwHbhqoqJth7WQYNe
lYCmOUYRaepxTrpU4TGDGuhqMh9D9\
noNMIFx3ktkcTitxkSY/5h/pXqSB2XXURLpZWwZxjEYwWCpWE8i7uVtIR7wIDAQAB\nAoGAOv0Kik9lOVbhsaK/yAO8w039d7l6h+NQaYtPbMhKzg4iofomp9DJBWK2a3sp\
nzoN4s9pTKFPmXC+1gb4sLULD72ENSgyozrPMCJ0tytNa3ebVCCYDvlagEZ83KwGn\nnr2BIRLul1xyHVa+amvTemuxi7OOx0u/0aZ6jPVW9ocPahkCQQDnUACG3Q2p60Mx\
nCttW7Go2wz0BhaI8ibG8rHorhdrFJUrsTI6QrLh340uHCIEAuWUXjYDX2u5MTPBG\njWbL/A/9AkEAjnX8EzO/B/RooFTxhYdxv7D1Dzcx4H59mFzjez6N/mjAHjKL4p66\
nqGFgLa9HMhDPFOs83VetBXcihu1vueffWwJBALQUD2TvAS0wz82FYz8nrITXuE3Q\nCH7Sv8FgEXiCq89hehO+ghrVrIMBPBJzJ2M18iLE8fKaKXzTRRfYC5hwss0CQHY8\
nBdIKCGoZtxwaY7lnCEkIHNtb+9FOKf7iWQpYiJC1b32ghei3xEMrTh+ccYJj4PqD\noigyNC9tCQLi3O92OjECQGYR2z8IDlfnl7G8p7eiyBciuoDyyq5/oJHhkHbkbwzr\
nW0Uun+rEcjRnXbUN8wUQ6FSFrxk2VSajbCBteTOrF24=\n-----END RSA PRIVATE KEY-----\n', u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'} >>> conn.ex_list_keypairs() [{u'name': u'test', u'fingerprint': u'43:59:7e:00:16:45:fc:ab:81:55:03:47:12:22:1e:d5'}, {u'name': u'foobar',
u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}] >>> conn.ex_delete_keypair('test') u'true' >>> conn.ex_list_keypairs() [{u'name': u'foobar', u'fingerprint': u'b9:2d:4b:07:db:e7:3e:42:17:11:22:33:44:55:66:77'}] >>> conn.ex_create_security_group('heyhey') {u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id':
u'77ad73b5-a383-4e13-94be-a38ef9877996', u'domain': u'runseb@gmail.com', u'name': u'heyhey'} >>> conn.ex_list_security_groups() [{u'egressrule': [], u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr': u'
0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description':
u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'},
{u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'id': u'77ad73b5-a383-4e13-94be-a38ef9877996',
u'domain': u'foobar@gmail.com', u'name': u'heyhey'}] >>> conn.ex_delete_security_group('heyhey') u'true' >>> conn.ex_list_security_groups() [{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
u'description': u'Default Security Group', u'tags': [], u'domain': u'foobar@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr':
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id': u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e',
u'name': u'default'}, {u'account': u'foobar@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
u'description': u'this is a test', u'domain': u'foobar@gmail.com', u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

With a security group created you can now add an ingress rule:

>>> conn.ex_authorize_security_group_ingress(securitygroupname='toto',protocol='TCP',cidrlist='0.0.0.0./0',
startport=99) {u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
u'description': u'this is a test', u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 99, u'cidr': u'0.0.0.0./0',
u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}],
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'} >>> conn.ex_list_security_groups() [{u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid': u'ab53d864-6f78-4993-bb28-9b8667b535a1',
u'description': u'Default Security Group', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule': [{u'startport': 22, u'cidr':
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 22, u'ruleid': u'b83428c0-7f4c-44d1-bc96-4e1720168fdf'}, {u'startport': 80, u'cidr':
u'0.0.0.0/0', u'protocol': u'tcp', u'endport': 80, u'ruleid': u'042124dd-652d-4fa2-8bee-d69973380f21'}], u'id':
u'ebfa2339-e9ae-4dcb-b73c-a76cd3fce39e', u'name': u'default'}, {u'egressrule': [], u'account': u'runseb@gmail.com', u'domainid':
u'ab53d864-6f78-4993-bb28-9b8667b535a1', u'description': u'this is a test', u'tags': [], u'domain': u'runseb@gmail.com', u'ingressrule':
[{u'startport': 99, u'cidr': u'0.0.0.0./0', u'protocol': u'tcp', u'endport': 99, u'ruleid': u'a13a21f9-1709-431f-9c7d-e1a2c2caacdd'}],
u'id': u'b7f5fbad-4244-491f-9547-c91a010e0c9d', u'name': u'toto'}]

Great, keypair and security group work fine. Now let's do the basics of starting an instance and let's test the suspend and resume which was not in the Driver until now. Nothing earth shattering but it's an improvement.

>>> size=conn.list_sizes()
>>> image=conn.list_images()

>>> n=conn.create_node(name='yoyo',size=size[0],image=image[0])

>>> n.ex_stop()
u'Stopped'
>>> n.ex_start()
u'Running'

And that's it for now, CloudStack support is getting better, there is still tons of work to do: improve the advanced zone support, check the load-balancer and storage support, add unit-tests and new CloudStack features like auto-scaling. Carry on !!!!

Build a Cloud Day Amsterdam recap

On June 13th we had a Build a Cloud Day workshop in Schuberg Phillis offices in Amsterdam. We had a good crowd with a little over 30 people gathering to learn about CloudStack, its use cases, monitoring, PaaS, SDN and client tools. We had good fun and it was a very productive meeting.

We started off with a talk from Arjan Eriks from Schuberg Phillis, Arjan gave us a terrific overview of the Schuberg's culture and how the Cloud and DevOps processes naturally appeared within Schuberg. Their team oriented approach to customer support evolved to using CloudStack as their core IT solution to provide an agile infrastructure that meets the need of their customers. Adding to the Cloud infrastructure, a DevOps culture permeated Schuberg even before DevOps really existed. His slides below will give you a good idea of how a company moves to the Cloud and uses agile methods to make customers and employees "happy".

Following Arjan we had the chance to hear from Julien Pivoto (@roidelapluie) of Inuits an IT consulting company that helps its customers adopt open source solutions and embrace a DevOps culture. Julien presented the suite of open source monitoring tools that can be used to monitor a Cloud infrastructure at scale. From logstash, elasticsearch and Kibana, to collectd, graphite and riemann. Julien did a great job introducing all these tools and explaining how they stitch together to help automation of a Cloud infrastructure.

Hugo Trippaers (@Spark404) then gave us a very practical talk about SDN (Software Defined Networking). Hugo has developed the Nicira NVP integraton in Apache CloudStack and manages the Cloud at Schuberg Phillis. He reviewed all the SDN support in CloudStack (e.g Nicira, native OVS manager, Big Switch, Midonet) and then dove into a Nicira use case. This really brought home the fact that SDN is not a fiction but a reality.

After these first talks about a company's view of Cloud and DevOps, a review of key monitoring tools and SDN solutions in CloudStack, Giles Sirett from Shapeblue presented us with several CloudStack use cases. Shapeblue is a CloudStack integrator in the UK with offices in India, US and Brazil. Giles insider's look on how company choose a Cloud solution and how CloudStack empowers them is unique. This was a very energizing talk showing use that CloudStack is gaining momentum and being adopted all over the world.

CloudStack is a IaaS solution, but once you have built your IaaS you need to focus on application delivery. Arash Kaffamanesh from Cloudssky talked to us about PaaS and showed a few demos on Appscale and Stackato integration with CloudStack. PaaS integration is set to become a hot topic within the next year and Arash lightning talk certainly set the stage. I know of folks working on better CloudStack support in Cloudfoundry/BOSH as well, so expect to see PaaS/CloudStack integration soon.

After these relatively strategic talk we went back to developer tools with Sander Botman. Sander is currently taking the lead in maintaing the Knife CloudStack plugin. Knife is the command line utility from Opscode, it is a great tool to manage your Chef repository. Written in ruby it is highly customizable and has numerous plugins. Sander demoed the knife-cloudstack plugin which let's you provision instances on your CloudStack based cloud and automatically boostrap these instances using chef recipes. I have used knife-cs recently and even submitted a few patches. It is a terrific tool, easy to use with lots of options. If you are looking for a tool to easily provision, configure and manage instances in your cloud, this is it.

Finally, I wrapped up the talks with demos about several CloudStack clients and tools. I have been testing lots of tools lately to prepare content for CloudStack University. I focused on Apache tools form the ASF such as libcloud and jclouds which just entered the incubator. I also looked at higher level wrappers like Apache whirr to create hadoop-clusters on demand. The slides below are still a work in progress and I will put up screencasts up really soon, Stay tuned.

To end the day, we had a few beers, before heading out to the DevOps days pre-conference party. A few folks sneaked out before we took the picture but we all left energized, having learned a ton. We all look forward to the CloudStack conference November 20-22nd in Amsterdam. Should be a blast.