Skip to content
This repository was archived by the owner on May 14, 2025. It is now read-only.

Commit c63fc35

Browse files
author
Corneil du Plessis
authored
Update ref docs (#5083, #4944) Added to Appendix - How to. (#5159)
1 parent 7bde74e commit c63fc35

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+2342
-60
lines changed

‎spring-cloud-dataflow-docs/src/main/asciidoc/appendix-create-containers.adoc‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
[[create-containers]]
2+
3+
=== Create containers for architectures not supported yet.
4+
In the case of macOS on M1 the performance of amd64/x86_64 is unacceptable.
5+
We provide a set of scripts that can be used to download specific versions of published artifacts.
6+
We also provide a script that will create a container using the downloaded artifact for the host platform.
7+
In the various projects you will find then in `src/local` or `local` folders.
8+
9+
[cols="1,4,6"]
10+
|===
11+
|Project | Scripts | Notes
12+
13+
| [.small]#Data Flow#
14+
a|
15+
[.small]#`src/local/download-apps.sh`# +
16+
[.small]#`src/local/create-containers.sh`# +
17+
a|[.small]#Download or create container for: `spring-cloud-dataflow-server`,# +
18+
[.small]#`spring-cloud-dataflow-composed-task-runner`,# +
19+
[.small]#`spring-cloud-dataflow-single-step-batch-job`,# +
20+
[.small]#`spring-cloud-dataflow-tasklauncher-sink-kafka`,# +
21+
[.small]#`spring-cloud-dataflow-tasklauncher-sink-rabbit`# +
22+
23+
|[.small]#Skipper#
24+
a|
25+
[.small]#`local/download-app.sh`# +
26+
[.small]#`local/create-container.sh`#
27+
|[.small]#Download or create container for: `spring-cloud-skipper-server`#
28+
29+
|[.small]#Stream Applications#
30+
a|
31+
[.small]#`local/download-apps.sh`# +
32+
[.small]#`local/create-containers.sh`# +
33+
[.small]#`local/pack-containers.sh`#
34+
| `create-containers.sh` uses `jib` +
35+
`pack-containers.sh` uses `pack`
36+
|===
37+
38+
==== Scripts in `spring-cloud-dataflow`
39+
===== `src/local/download-apps.sh`
40+
Downloads all applications needed by `create-containers.sh` from Maven repository.
41+
42+
*If the timestamp of snapshots matches the download will be skipped.*
43+
44+
Usage: `download-apps.sh [version]`
45+
46+
* `version` is the dataflow-server version like `2.9.6`. Default is `2.10.0-SNAPSHOT`
47+
48+
===== `src/local/create-containers.sh`
49+
Creates all containers and pushes to local docker registry.
50+
51+
This script requires link:https://github.com/GoogleContainerTools/jib/tree/master/jib-cli[jib-cli]
52+
53+
Usage: `create-containers.sh [version] [jre-version]`
54+
55+
* `version` is the dataflow-server version like `2.9.6`. Default is `2.10.0-SNAPSHOT`
56+
* `jre-version` should be one of 11, 17. Default is 11
57+
58+
==== Scripts in `spring-cloud-skipper`
59+
60+
===== `local/download-app.sh`
61+
Downloads all applications needed by `create-containers.sh` from Maven repository.
62+
63+
*If the timestamp of snapshots matches the download will be skipped.*
64+
65+
Usage: `download-app.sh [version]`
66+
67+
* `version` is the skipper version like `2.8.6` or default is `2.9.0-SNAPSHOT`
68+
69+
===== `local/create-container.sh`
70+
Creates all containers and pushes to local docker registry.
71+
This script requires link:https://github.com/GoogleContainerTools/jib/tree/master/jib-cli[jib-cli]
72+
73+
Usage: `create-containers.sh [version] [jre-version]`
74+
75+
* `version` is the skipper version like `2.8.6` or default is `2.9.0-SNAPSHOT`
76+
* `jre-version` should be one of 11, 17
77+
78+
==== Scripts in `stream-applications`
79+
80+
===== `local/download-apps.sh`
81+
82+
Downloads all applications needed by `create-containers.sh` from Maven repository.
83+
84+
*If the timestamp of snapshots matches the download will be skipped.*
85+
86+
Usage: `download-apps.sh [version] [broker] [filter]`
87+
88+
* `version` is the stream applications version like `3.2.1` or default is `3.2.2-SNAPSHOT`
89+
* `broker` is one of rabbitmq, rabbit or kafka
90+
* `filter` is a name of an application or a partial name that will be matched.
91+
92+
===== `local/create-containers.sh`
93+
Creates all containers and pushes to local docker registry.
94+
95+
This script requires link:https://github.com/GoogleContainerTools/jib/tree/master/jib-cli[jib-cli]
96+
97+
Usage: `create-containers.sh [version] [broker] [jre-version] [filter]`
98+
99+
* `version` is the stream-applications version like `3.2.1` or default is `3.2.2-SNAPSHOT`
100+
* `broker` is one of rabbitmq, rabbit or kafka
101+
* `jre-version` should be one of 11, 17
102+
* `filter` is a name of an application or a partial name that will be matched.
103+
104+
If the file is not present required to create the container the script will skip the one.
105+
106+
===== `local/pack-containers.sh`
107+
Creates all containers and pushes to local docker registry.
108+
109+
This script requires link:https://buildpacks.io/docs/tools/pack[packeto pack]
110+
111+
Usage: `pack-containers.sh [version] [broker] [jre-version] [filter]`
112+
113+
* `version` is the stream-applications version like `3.2.1` or default is `3.2.2-SNAPSHOT`
114+
* `broker` is one of rabbitmq, rabbit or kafka
115+
* `jre-version` should be one of 11, 17
116+
* `filter` is a name of an application or a partial name that will be matched.
117+
118+
If the required file is not present to create the container the script will skip that one.
119+
120+
NOTE: If any parameter is provided all those to the left of it should be considered required.

‎spring-cloud-dataflow-docs/src/main/asciidoc/appendix-development-tasks.adoc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[appendix]
2+
[[development-tasks]]
3+
== Development Tasks
4+
5+
include::appendix-extend-classpath.adoc[]
6+
include::appendix-create-containers.adoc[]
7+
include::appendix-local-k8s-development.adoc[]

‎spring-cloud-dataflow-docs/src/main/asciidoc/appendix-extend-classpath.adoc‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
[[extend-classpath]]
2+
3+
=== Extending container classpath
4+
5+
==== Containers
6+
7+
Users may require the addition of dependencies to the existing Stream applications or specific database drivers to Dataflow and Skipper or any of the other container provider by the project.
8+
9+
===== Instructions
10+
11+
In order to add dependencies to existing container the following steps will be the approach.
12+
13+
* Create a folder with the extra dependencies.
14+
* Create a new container image while copying the files to the libraries folder.
15+
* Push the image to a private registry.
16+
17+
===== Environmental variables
18+
19+
* `DEPS_FOLDER` should be a full filename or path expression for files to copy to the container.
20+
* `CONTAINER_REPO` the source docker image name.
21+
* `CONTAINER_TAG` the tag of source image.
22+
* `PRIVATE_REGISTRY` the host name of the private registry.
23+
24+
===== Examples
25+
26+
[source,shell]
27+
....
28+
export CONTAINER_REPO="springcloud/spring-cloud-dataflow-server"
29+
export CONTAINER_TAG="2.9.5-jdk17"
30+
export PRIVATE_REGISTRY="our.private.registry"
31+
export DEPS_FOLDER="./extra-libs/"
32+
docker build -t "$PRIVATE_REGISTRY/$CONTAINER_REPO:$CONTAINER_TAG"
33+
docker push "$PRIVATE_REGISTRY/$CONTAINER_REPO:$CONTAINER_TAG"
34+
....
35+
36+
==== JAR File
37+
38+
When using CloudFoundry or local deployment you will need to update jar before publishing it to a private registry or Maven Local.
39+
40+
===== Example
41+
42+
This example adds the dependencies and then installs the jar to Maven local.
43+
44+
[source,shell]
45+
....
46+
./gradlew -i publishToMavenLocal \
47+
-P appFolder="." \
48+
-P appGroup="org.springframework.cloud" \
49+
-P appName="spring-cloud-dataflow-server" \
50+
-P appVersion="2.9.5" \
51+
-P depFolder="./extra-libs"
52+
....
53+
54+
55+
56+
NOTE: Use the `publishMavenPublicationToMavenRepository` task to publish to a remote repository. Update the `gradle.properties` with the remote repository details. Alternatively move `repoUser` and `repoPassword` to ~/.gradle/gradle.properties
57+

‎spring-cloud-dataflow-docs/src/main/asciidoc/appendix-howto.adoc‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,6 @@ That is also a great place to ask new questions (use the `spring-cloud-dataflow`
99

1010
We are also more than happy to extend this section. If you want to add a "`how-to`", you can send us a {github-code}[pull request].
1111

12-
13-
1412
=== Configure Maven Properties
1513

1614
You can set the Maven properties, such as the local Maven repository location, remote Maven repositories, authentication credentials, and proxy server properties through command-line properties when you start the Data Flow server.
@@ -85,9 +83,14 @@ NOTE: Depending on the Spring Cloud Data Flow server implementation, you may hav
8583

8684
This section covers how to troubleshoot Spring Cloud Data Flow on your platform of choice. See the Troubleshooting sections of the microsite for link:https://dataflow.spring.io/docs/stream-developer-guides/troubleshooting/[Stream] and link:https://dataflow.spring.io/docs/batch-developer-guides/troubleshooting/[Batch] processing.
8785

86+
include::appendix-extend-classpath.adoc[]
87+
include::appendix-create-containers.adoc[]
88+
include::appendix-local-k8s-development.adoc[]
8889

8990
[[faqs]]
9091
=== Frequently Asked Questions
9192

9293
In this section, we review the frequently asked questions for Spring Cloud Data Flow.
9394
See the https://dataflow.spring.io/docs/resources/faq/[Frequently Asked Questions] section of the microsite for more information.
95+
96+
// TODO move the FAG to reference guide.

0 commit comments

Comments
 (0)