Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.
This article is Part 2 of a series on Cloud Foundry concepts:
- Deploying the spring-music webapp, Part 1
- Persisting spring-music data using Postgres service, Part 2
- Scaling the spring-music webapp, Part 3
- Logging for the spring-music webapp, Part 4
- Monitoring the spring-music webapp, Part 5
In this particular article, we will create a Cloud Foundry Postgres service to externalize the persistent store instead of using the default in-memory H2 database which is destroyed every time the application is restarted or restaged.
Previous articles in the series describe basic deployment of the spring-music application, and we pick up from there.
Default in-memory database
By default, the spring-music application uses an H2 in-memory database for persistence. This means it will only last as long as the JVM persists.
Let’s validate that the H2 database only persists until a restart or restage of the application. Pull up the application, and make an update to any of the albums.
Now restage the application, and pull up the application again.
$ cf restage spring-music
Notice that the change you made previously is gone. Make another change, and then restart the application:
$ cf restart spring-music
Once again the change is gone.
CF Service binding
If you want to maintain persistence across restaging, restarts, upgrades, and rolling restarts you are going to need a disk persisted database. This can be done by creating an externalized Database service, and binding this database service to the application.
We are going to dig down for second to explain how binding is accomplished. Each CF application has associated application metadata (VCAP_APPLICATION) and services metadata (VCAP_SERVICES). Here is a diagram from Jared Ruckle’s blog posting on the Pivotal blog:
For example, if you use ‘cf env’ on the default spring-music application with no bound services, you would see that it has only VCAP_APPLICATION metadata storing id, URL, resource limits, etc.
$ cf env spring-music
{
"VCAP_APPLICATION": {
"application_id": "0155d30f-2bcd-408d-b67b-887ee492307a",
"application_name": "spring-music",
"application_uris": [
"spring-music-undaunting-melanism.cfapps.io"
],
"application_version": "23479604-3a3e-4b87-9a0d-2f2a3fb291a7",
"cf_api": "https://api.run.pivotal.io",
"limits": {
"disk": 1024,
"fds": 16384,
"mem": 1024
},
"name": "spring-music",
"space_id": "a8b76dd2-323a-4ad8-80ae-8d098d4d17b2",
"space_name": "development",
"uris": [
"spring-music-undaunting-melanism.cfapps.io"
],
"users": null,
"version": "23479604-3a3e-4b87-9a0d-2f2a3fb291a7"
}
}
But, as an example, after binding a Postgres database to the service (just look for now, I show how to bind further down), there would also be a section called VCAP_SERVICES which has the Postgres parameters and tags that identify the bound service.
$ cf env spring-music
{
"VCAP_SERVICES": {
"elephantsql": [
{
"credentials": {
"max_conns": "5",
"uri": "postgres://nmthantb:EFCbWNVKhkzbf_tN48uBH5kp9pwb10KG@baasu-01.db.elephantsql.com:5432/nmthantb"
},
"label": "elephantsql",
"name": "spring-music-db",
"plan": "turtle",
"provider": null,
"syslog_drain_url": null,
"tags": [
"Data Stores",
"Web-based",
"Open Source",
"User Provisioning",
"PaaS",
"Single Sign-On",
"Windows",
"New Product",
"Mac",
"Android",
"Developer Tools",
"Data Store",
"postgresql",
"Buyable",
"relational",
"Importable",
"IT Management"
],
"volume_mounts": []
}
]
}
}
{
"VCAP_APPLICATION": {
"application_id": "0155d30f-2bcd-408d-b67b-887ee492307a",
"application_name": "spring-music",
"application_uris": [
"spring-music-undaunting-melanism.cfapps.io"
],
"application_version": "23479604-3a3e-4b87-9a0d-2f2a3fb291a7",
"cf_api": "https://api.run.pivotal.io",
"limits": {
"disk": 1024,
"fds": 16384,
"mem": 1024
},
"name": "spring-music",
"space_id": "a8b76dd2-323a-4ad8-80ae-8d098d4d17b2",
"space_name": "development",
"uris": [
"spring-music-undaunting-melanism.cfapps.io"
],
"users": null,
"version": "23479604-3a3e-4b87-9a0d-2f2a3fb291a7"
}
}
This is agnostic of whether you use Java, Java Spring, Ruby, or Node.js. VCAP_SERVICES will be loaded with metadata about the bound services.
How each platform interprets VCAP_SERVICES, parses it, and then enables the select services is entirely up to the framework and programmer.
Creating a Database Service
Let’s go ahead and create a persistent database service and attach it to our application. The CloudFoundry marketplace contains a Postgres database from ElephantSQL that makes it easy to deploy into our space.
Search the marketplace, and you will find the elephantsql ‘turtle’ service plan is free.
$ cf marketplace -s elephantsql Getting service plan information for service elephantsql as fabian.lee@test1.com... OK service plan description free or paid turtle 4 concurrent connections, 20MB Storage free panda 20 concurrent connections, 2GB Storage paid hippo 300 concurrent connections, 100 GB Storage paid elephant 300 concurrent connections, 1000 GB Storage, 500Mbps paid
Now create the database service, and then list all the services we have created in our space.
> cf create-service elephantsql turtle spring-music-db Creating service instance spring-music-db in org fleetest1 / space development as fabian.lee@test1.com... OK > cf services Getting services in org fleetest1 / space development as fabian.lee@test1.com... OK name service plan bound apps last operation spring-music-db elephantsql turtle create succeeded
It shows a brief listing which confirms it has been created. But if you want more details on the particular service, such as the URL for the administrative dashboard of the database:
$ cf service spring-music-db Service instance: spring-music-db Service: elephantsql Bound apps: spring-music Tags: Plan: turtle Description: PostgreSQL as a Service Documentation url: http://docs.run.pivotal.io/marketplace/services/elephantsql.html Dashboard: https://cloudfoundry.appdirect.com/api/custom/cloudfoundry/v2/sso/start?serviceUuid=1b89fed5-704d-40fb-856a-3acc0c6d3616 Last Operation Status: create succeeded Message: Started: 2017-11-10T00:47:13Z Updated: 2017-11-10T00:47:13Z
Then bind the ‘spring-music-db’ service to the ‘spring-music’ app, and restage so these changes take effect:
> cf bind-service spring-music spring-music-db Binding service spring-music-db to app spring-music in org fleetest1 / space development as fabian.lee@test1.com... OK TIP: Use 'cf restage spring-music' to ensure your env variable changes take effect > cf restage cf-music
Pull up the application at the new URL and make a change to one of the albums. Then restart the application:
$ cf restart spring-music
When you pull up the new URL, notice that the change persisted. As further proof to yourself, delete the application entirely (but not the database service) and create a new app instance bound to the same database service.
$ cf delete spring-music -f $ cf push spring-music --no-stop $ cf bind-service spring-music spring-music-db $ cf start spring-music
Even though the application will be at a new URL, the change you made to the album should still be reflected in the interface.
CF Service binding with Spring
If you are not a developer, you can skip this last section, but what I have not yet described is how Spring itself handles the service binding.
I’ll go over three of the primary methods, with the understanding this is a shallow explanation that you’ll need to dig into if you are designing a production solution [1,2,3,4,5,6].
The cf-db-services-sample project is a good source code example illustrating the methods below.
Java Buildpack Auto-Reconfiguration
Buildpack auto-reconfiguration is the most magical way of binding a service. It adds the “cloud” active profile to Spring, exposes the CF environment variables as a PropertySource, and then rewrites Beans of certain types and binds them to the service.
For example, any object of type javax.sql.DataSource is bound to a relational database, and a type of org.springframework.data.mongodb.MongoDbFactory is bound to Mongo.
But this approach typically isn’t rich enough to handle production needs.
Spring Boot DataSource using properties
Spring Boot allows you to specify a DataSource using properties found in application.properties/yml. This makes the implementation look similar to any other Spring Boot application.
The properties passed in via the VCAP_SERVICES json structure are flattened out and can be put into a propeties/yaml file that does substitution so that Spring Boot loads it like any standard data source.
spring:
datasource:
url: ${vcap.services.mydb.credentials.jdbcUrl}
username: ${vcap.services.mydb.credentials.username}
password: ${vcap.services.mydb.credentials.password}
Spring Cloud Connectors
The spring-music project uses the Spring Cloud Cloud Foundry Connector and Spring Cloud Service Connector which is a powerful way of injecting the correct services.
A DataSource or Factory can be retrieved by extending the AbstractCloudConfig class as illustrated in RelationalCloudDataSourceConfig.java:
@Configuration
@Profile({"mysql-cloud", "postgres-cloud", "oracle-cloud", "sqlserver-cloud"})
public class RelationalCloudDataSourceConfig extends AbstractCloudConfig {
@Bean
public DataSource dataSource() {
return connectionFactory().dataSource();
}
}
The Cloud Connector knows a DataSource is required, and looking at the tags in VCAP_SERVICES knows that the service with a “postgres” tag is a candidate.
The @Profile annotation is way to conditionally activate components, based on configuration. The active profile set is built by the Cloud Connector and CF Cloud Connector. For example, “SpringApplicationContextInitializer.java” uses the Cloud Connector API to retrieve the services and then output the active profiles.
List<ServiceInfo> serviceInfos = cloud.getServiceInfos();
logger.info("Found serviceInfos: " + StringUtils.collectionToCommaDelimitedString(serviceInfos));
.
.
.
logger.info("Setting profile names: " + StringUtils.arrayToCommaDelimitedString(profileNames));
Below is the log snippet from the two log lines above showing how Postgres is in the active profile (which excludes the beans for Redis, Mongo, and other data sources).
2017-11-13T18:32:49.77-0500 [APP/PROC/WEB/0] OUT 17-November-13 23:32:49:772 INFO main o.c.s.m.c.SpringApplicationContextInitializer:68 - Found serviceInfos: PostgresqlServiceInfo[postgres://nmthantb:****@baasu-01.db.elephantsql.com:5432/nmthantb] . . . 2017-11-13T18:32:49.77-0500 [APP/PROC/WEB/0] OUT 17-November-13 23:32:49:773 INFO main o.c.s.m.c.SpringApplicationContextInitializer:127 - Setting profile names: postgres,postgres-cloud
Using this method, the spring-music project is capable of binding to local or cloud services and both traditional DataSource as well as factories provided by Mongo and Redis.
REFERENCES
https://spring.io/blog/2015/04/27/binding-to-data-services-with-spring-boot-in-cloud-foundry (under the covers how datasource selection is happening)
https://cloud.spring.io/spring-cloud-connectors/ (Spring Cloud Connectors)
https://cloud.spring.io/spring-cloud-connectors/spring-cloud-cloud-foundry-connector.html (Spring CF cloud connector)
https://spring.io/blog/2011/10/13/using-cloud-foundry-services-with-spring-part-1-the-basics/ (using spring-hello-env to show props)
https://spring.io/blog/2011/11/10/using-cloud-foundry-services-with-spring-part-4-spring-profiles
http://www.java-allandsundry.com/2016/05/approaches-to-binding-spring-boot.html (how VCAP_SERVICES json is translated to flat properties by Spring Boot, and therefore can be put into application.yml)
https://libraries.io/github/ebornier-pivotal/CloudNativeTour (multiple labs, debug endpoints like /env, /autoconfig, /configprops, /beans, /mappings, /dump, /trace, /info)
https://github.com/cloudfoundry/cf-mysql-deployment (mysql with persistent disks)
https://content.pivotal.io/blog/cloud-foundry-brazen-opinions-and-easy-extensions (service brokers)
http://dojoblog.emc.com/category/persistence/ (persistence on CF with volume mounts)
https://github.com/cloudfoundry/java-buildpack/issues/77 (auto reconfiguration should not happen when cloud is enabled, but in case)
NOTES
cf env spring-music (will show changes in bound databases)
(for mongo testing)
cf create-service mlab sandbox spring-music-mongodb
cf bind-service spring-music spring-music-mongodb
cf env spring-music