Preface ...................................................... xvii
Acknowledgments ............................................. xxiii
PART I Introduction to cloud computing for geosciences ......... 1
1 Geoscience application challenges to computing
infrastructures ............................................ 3
1.1 Challenges and opportunities for geoscience applications
in the 21st century ........................................ 3
1.1.1 Energy .............................................. 3
1.1.2 Emergency response .................................. 4
1.1.3 Climate change ...................................... 5
1.1.4 Sustainable development ............................. 6
1.2 The needs of a new computing infrastructure ................ 8
1.2.1 Providing enough computing power .................... 8
1.2.2 Responding in real time ............................. 9
1.2.3 Saving energy ....................................... 9
1.2.4 Saving the budget .................................. 10
1.2.5 Improving accessibility ............................ 10
1.3 The birth of cloud computing .............................. 10
1.3.1 Distributed computing .............................. 10
1.3.2 On-demand services ................................. 11
1.3.3 Computing sharing and cost savings ................. 12
1.3.4 Reliability ........................................ 12
1.3.5 The emergence of cloud computing ................... 12
1.4 The advantages and disadvantages of cloud computing for
geoscience applications ................................... 14
1.4.1 Advantages of cloud computing ...................... 14
1.4.2 Problems ........................................... 14
1.5 Summary ................................................... 15
1.6 Problems .................................................. 15
References ................................................ 16
2 Cloud computing architecture, concepts, and
characteristics ........................................... 19
2.1 Concepts .................................................. 19
2.2 Cloud computing architecture .............................. 20
2.3 Characteristics ........................................... 22
2.4 Service models ............................................ 23
2.5 Deployment models and cloud types ......................... 25
2.6 Review of cloud computing resources ....................... 27
2.6.1 Commercial clouds .................................. 27
2.6.2 Open-source cloud solutions ........................ 29
2.7 Summary ................................................... 30
2.8 Problems .................................................. 30
References ................................................ 31
3 Enabling technologies ..................................... 33
3.1 Hardware advancements ..................................... 33
3.1.1 Multicore and many-core technologies ............... 33
3.1.2 Networking ......................................... 34
3.1.3 Storage ............................................ 34
3.1.4 Smart devices ...................................... 35
3.2 Computing technologies .................................... 35
3.2.1 Distributed computing paradigm ..................... 35
3.2.2 Computing architecture model ....................... 36
3.3 Virtualization ............................................ 36
3.3.1 Virtualization implementation ...................... 37
3.3.2 Virtualization solutions ........................... 38
3.4 Distributed file system ................................... 39
3.4.1 Introduction to the distributed file system ........ 40
3.4.2 Google File System ................................. 40
3.4.3 Apache Hadoop Distributed File System .............. 41
3.5 Web x.0 ................................................... 42
3.5.1 Web services ....................................... 43
3.5.2 Service-oriented architecture ...................... 44
3.6 Conclusion ................................................ 46
3.7 Summary ................................................... 46
3.8 Problems .................................................. 47
References ................................................ 47
PART II Deploying applications onto cloud services ............. 49
4 How to use cloud computing ................................ 51
4.1 Popular cloud services .................................... 51
4.1.1 Introduction ....................................... 51
4.1.2 Amazon AWS and Windows Azure ....................... 52
4.2 Use case: A simple Web Application ........................ 53
4.2.1 HTML design for the Hello Cloud Web application .... 53
4.2.2 Web servers ........................................ 54
4.3 Deploying the Web application onto cloud services ......... 55
4.3.1 Amazon Web Services ................................ 55
4.3.2 Windows Azure ...................................... 66
4.4 Conclusion and discussion ................................. 70
4.5 Summary ................................................... 72
4.6 Problems .................................................. 72
References ................................................ 73
5 Cloud-enabling geoscience applications .................... 75
5.1 Common components for geoscience applications ............. 75
5.1.1 Server-side programming ............................ 75
5.1.2 Database ........................................... 76
5.1.3 High performance computing ......................... 76
5.2 Cloud-enabling geoscience applications .................... 77
5.3 Use cases ................................................. 78
5.3.1 Database-driven Web applications ................... 78
5.3.2 Typical HPC applications ........................... 84
5.4 Summary ................................................... 90
5.5 Problems .................................................. 90
References ................................................ 90
6 How to choose cloud services: Toward a cloud computing
cost model ................................................ 93
6.1 The importance and challenges of selecting cloud
services .................................................. 93
6.2 The factors impacting cloud service selection ............. 94
6.2.1 Cloud service capacity provisioning and
measurements ....................................... 94
6.2.2 Cloud platform pricing rules ....................... 96
6.2.3 Application features and requirements .............. 97
6.3 Selecting cloud services using the Earth Science
Information Partners (ESIP) cloud Adoption Advisory Tool
as an example ............................................. 98
6.3.1 Architecture of the advisory tool .................. 99
6.3.2 The general workflow for cloud service selection ... 99
6.3.3 Use case .......................................... 102
6.4 In-depth considerations in cloud service selection
and the development of advisory tools .................... 104
6.4.1 Correctness and accuracy of evaluation models ..... 105
6.4.2 Up-to-date information of cloud services .......... 106
6.4.3 Interactivity and visualization functions of the
advisory tool ..................................... 106
6.5 Summary .................................................. 106
6.6 Problems ................................................. 107
References ............................................... 107
PART III Cloud-enabling geoscience projects ................... 109
7 ArcGIS in the cloud ...................................... 111
7.1 Introduction ............................................. 111
7.1.1 Why a geographical information system needs the
cloud ............................................. 111
7.1.2 GIS examples that need the cloud .................. 112
7.2 ArcGIS in the cloud ...................................... 112
7.2.1 ArcGIS Online ..................................... 112
7.2.1.1 Functionalities ............................... 113
7.2.2 ArcGIS for Server ................................. 114
7.2.2.1 Functionalities ............................... 114
7.2.3 GIS software as a service ......................... 115
7.2.3.1 Functionalities ............................... 115
7.2.4 Mobile GIS service ................................ 116
7.2.5 Section summary ................................... 116
7.3 Use cases ................................................ 117
7.3.1 Regional analysis of Oregon using ArcGIS Online ... 117
7.3.2 Use cases of ArcGIS for Server .................... 120
7.3.2.1 Brisbane City Council Flood Common
Operating Picture ............................. 120
7.3.2.2 Pennsylvania State Parks Viewer ............... 121
7.3.3 Section summary ................................... 122
7.4 Summary .................................................. 122
7.5 Problems ................................................. 123
References ............................................... 124
8 Cloud-enabling GEOSS Clearinghouse ....................... 125
8.1 GEOSS Clearinghouse: Background and challenges ........... 125
8.1.1 Background ........................................ 125
8.1.2 Challenges ........................................ 126
8.2 Deployment and optimization .............................. 127
8.2.1 General deployment workflow ....................... 127
8.2.2 Special considerations ............................ 130
8.2.2.1 Data backup .............................. 130
8.2.2.2 Load balancing ........................... 132
8.2.2.3 Auto-scaling ............................. 132
8.2.3 The differences from the general steps in
Chapter 5 ......................................... 134
8.3 System demonstration ..................................... 135
8.3.1 Local search ...................................... 135
8.3.2 Remote search ..................................... 136
8.4 Conclusion ............................................... 136
8.4.1 Economic advantages ............................... 137
8.4.2 Technical advantages .............................. 138
8.5 Summary .................................................. 138
8.6 Problems ................................................. 138
Appendix 8.1 Template for creating an auto-scaling
function ................................................. 138
References ............................................... 141
9 Cloud-enabling Climate@Home .............................. 143
9.1 Climate@Home: Background and challenges .................. 143
9.1.1 Background ........................................ 143
9.1.2 Challenges ........................................ 146
9.2 Deployment and optimizations ............................. 147
9.2.1 General deployment workflow ....................... 147
9.2.1.1 Deploying the spatial Web portal ......... 147
9.2.1.2 Deploying the BOINC server ............... 149
9.2.2 Special considerations ............................ 151
9.2.3 The differences from the general steps in
Chapter 5 ......................................... 152
9.3 System demonstrations .................................... 153
9.3.1 Overview of the spatial Web portal ................ 153
9.3.2 The geovisual analytical portlet .................. 154
9.3.3 The resource management portlet ................... 155
9.4 Conclusion ............................................... 155
9.5 Summary .................................................. 157
9.6 Problems ................................................. 158
Appendix 9.1 E4M20a_000040_wu.xml ........................ 158
Appendix 9.2 Climateathome_re.xml ........................ 159
References ............................................... 159
10 Cloud-enabling dust storm forecasting .................... 161
10.1 Dust storm modeling: Background and challenges ........... 161
10.1.1 Background ........................................ 161
10.1.2 Challenges ........................................ 162
10.2 Deployment and optimization .............................. 164
10.2.1 General workflow .................................. 164
10.2.2 Special considerations ............................ 167
10.2.3 Summary of the differences from the general
steps in Chapter 5 ................................ 169
10.3 Demonstration ............................................ 169
10.3.1 Phoenix dust storm event .......................... 169
10.3.2 Simulation result ................................. 170
10.3.3 Performance ....................................... 170
10.3.4 Cost-efficiency ................................... 173
10.4 Conclusion ............................................... 174
10.5 Summary .................................................. 175
10.6 Problems ................................................. 175
References ............................................... 175
PART IV Cloud computing status and readiness .................. 177
11.1 Introduction to cloud services ........................... 179
11.1.1 Geographic presence ............................... 179
11.1.2 User interfaces and access to servers ............. 180
11.1.3 Automatic scaling and load balancing .............. 180
11.1.4 Service Level Agreement (SLA) ..................... 180
11.2 Amazon Web Services (AWS) ................................ 181
11.2.1 Architecture ...................................... 181
11.2.2 General characteristics of EC2 .................... 182
11.2.2.1 Scalability ................................... 182
11.2.2.2 Compatibility ................................. 183
11.2.2.3 Deployment and interface ...................... 183
11.2.2.4 Hypervisors ................................... 183
11.2.2.5 Reliability ................................... 183
11.2.2.6 OS supports ................................... 184
11.2.2.7 Cost .......................................... 184
11.2.3 Major users and general comments .................. 185
11.2.3.1 List of major customers ....................... 185
11.2.3.2 Maturation .................................... 186
11.2.3.3 Feedback from the community ................... 186
11.2.4 Usage complexity .................................. 187
11.3 Windows Azure ............................................ 188
11.3.1 Architecture ...................................... 188
11.3.2 General characteristics of Windows Azure .......... 189
11.3.2.1 Scalability ................................... 189
11.3.2.2 Compatibility ................................. 190
11.3.2.3 Deployment and interface ...................... 191
11.3.2.4 Hypervisors ................................... 191
11.3.2.5 Reliability ................................... 191
11.3.2.6 OS supports ................................... 191
11.3.2.7 Cost .......................................... 191
11.3.3 Major users and general comments .................. 192
11.3.3.1 List of major customers ....................... 192
11.3.3.2 Maturation .................................... 193
11.3.3.3 Feedback from community ....................... 193
11.3.3.4 Usage complexity .............................. 193
11.4 Nebula ................................................... 194
11.4.1 Architecture ...................................... 194
11.4.2 General characteristics of Nebula ................. 195
11.4.2.1 Scalability ................................... 195
11.4.2.2 Compatibility ................................. 195
11.4.2.3 Deployment and interface ...................... 195
11.4.2.4 Hypervisors ................................... 195
11.4.2.5 Reliability ................................... 195
11.4.2.6 OS supports ................................... 195
11.4.2.7 Cost .......................................... 195
11.4.3 Major users and general comments .................. 196
11.4.3.1 List of major customers ....................... 196
11.4.3.2 Maturation .................................... 196
11.4.3.3 Feedback from community ....................... 196
11.4.3.4 Usage complexity .............................. 196
11.5 Conclusion .......................................... 196
11.6 Summary ............................................. 197
11.7 Problems ............................................ 198
References ............................................... 198
12 How to test the readiness of cloud services .............. 201
12.1 Introduction ............................................. 201
12.2 Test environment ......................................... 201
12.2.1 Network ........................................... 201
12.2.2 Computing service configuration ................... 202
12.3 Concurrent intensity test using GEOSS Clearinghouse
(CLH) .................................................... 202
12.3.1 Clearinghouse requirements for computing
services .......................................... 202
12.3.2 Test design ....................................... 203
12.3.3 Test workflow ..................................... 204
12.3.4 Test result analyses .............................. 206
12.4 Data and computing intensities test using Climate@Home ... 207
12.4.1 Climate@Home computing requirements ............... 207
12.4.2 Test design ....................................... 208
12.4.3 Test workflow ..................................... 208
12.4.4 Test result analyses .............................. 211
12.5 Cloud test using dust storm forecasting .................. 211
12.5.1 Dust storm forecasting computing requirements ..... 211
12.5.2 Test design ....................................... 212
12.5.3 Test workflow ..................................... 213
12.5.4 Test result analyses .............................. 215
12.6 Summary .................................................. 216
12.7 Problems ................................................. 217
Appendix 12.1: GetRecords example to search metadata ..... 218
Appendix 12.2: Example of Jmeter test plan ............... 218
References ............................................... 221
13 Open-source cloud computing solutions .................... 223
13.1 Introduction to open-source cloud computing solutions .... 223
13.1.1 CloudStack ........................................ 224
13.1.2 Eucalyptus ........................................ 224
13.1.3 Nimbus ............................................ 225
13.1.4 OpenNebula ........................................ 225
13.2 CloudStack ............................................... 225
13.2.1 Architecture ...................................... 225
13.2.2 General characteristics of CloudStack ............. 226
13.2.3 Major users and general comments on the
CloudStack solution ............................... 227
13.3 Eucalyptus ............................................... 228
13.3.1 Architecture ...................................... 228
13.3.2 General characteristics of Eucalyptus ............. 229
13.3.3 Major users and general comments on Eucalyptus .... 230
13.4 OpenNebula ............................................... 231
13.4.1 Architecture ...................................... 231
13.4.2 General characteristics of OpenNebula ............. 231
13.4.3 Major users and general comments on OpenNebula .... 233
13.5 Nimbus ................................................... 234
13.5.1 Architecture ...................................... 234
13.5.2 General characteristics of Nimbus ................. 235
13.5.3 Major users and general comments on Nimbus ........ 235
13.6 Open-source benchmarking considerations .................. 236
13.7 Conclusion ............................................... 236
13.8 Summary .................................................. 236
13.9 Problems ................................................. 239
References .................................................... 239
14 How to test the readiness of open-source cloud computing
solutions ................................................ 241
14.1 Introduction ............................................. 241
14.2 Test environment ......................................... 241
14.3 Tests of cloud operations ................................ 243
14.4 Tests of virtual computing resources ..................... 244
14.4.1 Brief introduction ................................ 244
14.4.2 Test design ....................................... 245
14.4.3 Test workflow ..................................... 246
14.4.4 Test result analyses .............................. 249
14.5 Tests of general applications ............................ 251
14.5.1 Brief introduction of test aspects ................ 251
14.5.2 Test design ....................................... 251
14.5.3 Testing workflow .................................. 252
14.5.4 Test result analyses .............................. 253
14.6 Cloud readiness test for GEOSS Clearinghouse ............. 254
14.6.1 Clearinghouse computing requirements .............. 254
14.6.2 Test design, workflow, and analysis ............... 254
14.7 Cloud readiness test for dust storm forecasting .......... 255
14.7.1 Dust storm forecasting computing requirements ..... 255
14.7.2 Test design ....................................... 255
14.7.3 Test workflow ..................................... 256
14.7.4 Test result analyses .............................. 256
14.8 Summary .................................................. 256
14.9 Problems ................................................. 257
Appendix 14.1 Ubench Patch File (Ubench-Patch.Txt) ....... 257
References .................................................... 258
15 GeoCloud initiative ...................................... 261
15.1 Introduction ............................................. 261
15.2 GeoCloud architecture .................................... 262
15.3 GeoCloud activities ...................................... 264
15.3.1 Creating prototype platforms ...................... 265
15.3.2 Validate with agency applications ................. 267
15.3.3 Document and promulgate ........................... 267
15.4 GeoCloud security ........................................ 268
15.4.1 Amazon Web Services (AWS) Security ................ 268
15.4.2 GeoCloud security operation ....................... 269
15.5 Operational cost in the cloud ............................ 269
15.6 Discussion ............................................... 270
15.7 Summary .................................................. 271
15.8 Problems ................................................. 271
References ............................................... 271
PART V Future directions ..................................... 273
16 Handling intensities of data, computation, concurrent
access, and spatiotemporal patterns ...................... 275
16.1 Introduction ............................................. 275
16.2 Big data ................................................. 276
16.2.1 Introduction ...................................... 276
16.2.2 An example with Climate@Home....................... 277
16.2.3 Solutions ......................................... 277
16.2.4 Remaining problems and future research ............ 278
16.3 Computing intensity ...................................... 280
16.3.1 Introduction ...................................... 280
16.3.2 An example with digital evaluation model
interpolation ..................................... 280
16.3.3 Solutions ......................................... 281
16.3.3.1 Cloud CPU computing ........................... 281
16.3.3.2 Cloud GPU computing ........................... 282
16.3.4 Remaining problems and future research ............ 283
16.4 Concurrent intensity ..................................... 284
16.4.1 Introduction ...................................... 284
16.4.2 An example with GEOSS Clearinghouse ............... 285
16.4.3 Solution .......................................... 286
16.4.3.1 Global content delivery ....................... 286
16.4.3.2 Elasticity .................................... 287
16.4.3.3 Spatiotemporal indexing ....................... 287
16.4.4 Remaining problems and future research ............ 288
16.5 Spatiotemporal intensity ................................. 289
16.6 Summary .................................................. 290
16.7 Problems ................................................. 290
References ............................................... 291
17 Cloud computing research for geosciences and
applications ............................................. 295
17.1 Evolving 21st century vision for geoscience
applications ............................................. 295
17.1.1 Fundamental geospatial science inquiries .......... 295
17.1.2 Integrating geoscience with other domains of
science for new discoveries ....................... 296
17.1.3 Application vision ................................ 296
17.2 Technological advancements ............................... 297
17.2.1 Cloud evaluation and selection .................... 297
17.2.2 Cloud service resource management ................. 298
17.2.3 Data backup and synchronization ................... 299
17.2.4 Interoperability .................................. 299
17.2.5 New visualization and interactive systems ......... 301
17.2.6 Reliability and availability ...................... 302
17.2.7 Real-time simulation and access ................... 302
17.3 Synergistic advancement of social science and cloud
computing ................................................ 303
17.3.1 Cloud management .................................. 303
17.3.2 Cloud outreach .................................... 303
17.3.3 Security and regulations .......................... 304
17.3.4 Global collaboration .............................. 305
17.4 Summary .................................................. 305
17.5 Problems ................................................. 306
References ............................................... 306
17.4 Index .................................................... 311
|
