docker registry_v2 部署及错误总结


docker registry_v2

docker registry_v2的搭建和排错文档,nginx+registry源码搭建,有别于网上类docker的搭建方法,方便registry日后调优

搭建过程

  • CA证书的制作(openssl)
  • nginx的搭建及配置
  • registry源码编译及配置
  • 验证及排错


CA证书的制作

1.首先我们去/etc/ssl/openssl.cnf下修改下参数,必须在生成证书之前修改,否则无意义

[ CA_default ]
    dir             = /etc/ssl/demoCA       # Where everything is kept
    certs           = $dir/certs            # Where the issued certs are kept
    crl_dir         = $dir/crl              # Where the issued crl are kept
    database        = $dir/index.txt        # database index file.
    #unique_subject = no                    # Set to 'no' to allow creation of
                                            # several ctificates with same subject.
    new_certs_dir   = $dir/newcerts         # default place for new certs.

    certificate     = $dir/certs/cacert.pem         # The CA certificate
    serial          = $dir/serial           # The current serial number
    crlnumber       = $dir/crlnumber        # the current crl number
                                            # must be commented out to leave a V1 CRL
    crl             = $dir/crl.pem          # The current CRL
    private_key     = $dir/private/cakey.pem# The private key
    RANDFILE        = $dir/private/.rand    # private random number file

    [ v3_req ]

    # Extensions to add to a certificate request

    basicConstraints = CA:FALSE

    keyUsage = nonRepudiation, digitalSignature, keyEncipherment

    #这个很重要,否则在后面会报registry endpoint...x509: cannot validate certificate for ... because it doesn't contain any IP SANs
    subjectAltName=IP:192.168.172.150


2.制作证书

证书的配置文件都在Ubuntu的路径在/etc/ssl下
cd /etc/ssl
    mkdir demoCA demoCA/certs demoCA/crl demoCA/newcerts demoCA/private
    touch /etc/ssl/demoCA/index.txt
    echo 01 > /etc/ssl/demoCA/serial
    cd /etc/ssl/demoCA
    openssl req -newkey rsa:4096 -nodes -sha256 -keyout cakey.pem -x509 -days 365 -out cacert.pem
    mv cacert.pem certs/ && mv cakey.pem private/


注意这里的domain设置成自己的域名即可,比如我的是*.192.168.172.150.xip.io

You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [XX]:CN
    State or Province Name (full name) []:beijing
    Locality Name (eg, city) [Default City]:beijing
    Organization Name (eg, company) [Default Company Ltd]:self
    Organizational Unit Name (eg, section) []:
    Common Name (eg, your name or your server's hostname) []:*.192.168.172.150.xip.io
    Email Address []:jackyuan@126.com


OK,至此,根证书等制作完成

nginx的搭建及配置

1.选择版本安装,最好是高版本,否则add header功能没法使用

cd ~
    wget http://nginx.org/download/nginx-1.9.4.tar.gz
    tar zxvf nginx-1.9.4.tar.gz
    cd ./nginx-1.9.4 && \
    ./configure --user=www --group=www --prefix=/opt/nginx \
    --with-pcre \
    --with-http_stub_status_module \
    --with-http_ssl_module \
    --with-http_addition_module  \
    --with-http_realip_module \
    --with-http_flv_module && \
    make && \
    make install


2.生成nginx的ssl证书,并加入进openssl本身的证书数据库

mkdir -p /etc/nginx/ssl
    cd /etc/nginx/ssl
    openssl genrsa -out nginx.key 4096

    openssl req -new -key nginx.key -out nginx.csr
    #上面这一步的配置要和跟设置的一样,尤其是domain那块
    openssl ca -in nginx.csr -out nginx.crt


在这里如果不在之前配置好CA的配置,则会出现demoCA无法打开等错误,所以要注意。

3.生成htpassword,用户名和密码都为admin
htpasswd -cb /opt/nginx/conf/.htpasswd admin admin


4.修改nginx配置

user  www www;
    worker_processes  auto;

    error_log   /var/log/nginx_error.log error;
    #error_log  logs/error.log  notice;
    #error_log  logs/error.log  info;

    #pid        logs/nginx.pid;


    worker_rlimit_nofile 51200;

    events {
        use epoll;
        worker_connections  51200;
        multi_accept on;
    }

    http {
        include       mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx_access.log  main;

        server_names_hash_bucket_size 128;
        client_header_buffer_size 32k;
        large_client_header_buffers 4 32k;

        sendfile        on;
        tcp_nopush     on;
        tcp_nodelay    on;

        #keepalive_timeout  0;
        keepalive_timeout  65;

        #gzip  on;

        upstream registry {
            server 192.168.172.150:5000;
        }    

        server {
            listen       443;
            server_name  192.168.172.150;

            ssl          on;
            ssl_certificate /etc/nginx/ssl/nginx.crt;
            ssl_certificate_key /etc/nginx/ssl/nginx.key;

            client_max_body_size 0;

            chunked_transfer_encoding on;

            location /v2/ {
              auth_basic "Registry realm";
              auth_basic_user_file /opt/nginx/conf/.htpasswd;
              add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

              proxy_pass                          http://registry;
              proxy_set_header  Host              \$http_host;   # required for docker client's sake
              proxy_set_header  X-Real-IP         \$remote_addr; # pass on real client's IP
              proxy_set_header  X-Forwarded-For   \$proxy_add_x_forwarded_for;
              proxy_set_header  X-Forwarded-Proto $scheme;
              proxy_read_timeout                  900;
            }

    #        location / {
    #            auth_basic "docker registry";
    #            auth_basic_user_file /opt/nginx/conf/.htpasswd;
    #            
    #            root   html;
    #            index  index.html index.htm;
    #
    #            proxy_pass   http://registry;
    #            proxy_set_header Host $http_host;
    #            proxy_set_header X-Real-IP $remote_addr;
    #            proxy_set_header Authorization "";

    #            client_body_buffer_size 128k;
    #            proxy_connect_timeout   90;
    #            proxy_send_timeout      90;
    #            proxy_read_timeout      90;
    #            proxy_buffer_size       8k;
    #            proxy_buffers        4 32k;
    #            proxy_busy_buffers_size 64k;
    #            proxy_temp_file_write_size 64k;
    #        }

    #        location /_ping {
    #         auth_basic off;
    #          proxy_pass http://registry;
    #        }

    #        location /v1/_ping {
    #         auth_basic off;
    #          proxy_pass http://registry;
    #        }

            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            }
        }
    
}

这里要说明两个地方:proxy_set_header X-Forwarded-Proto $scheme;注意这里没有\$scheme,如果配置则会报:unsupported protocol scheme ""

PS:认证是LDAP的可以在nginx里设置一下

user  www www;
    worker_processes  auto;

    error_log   /var/log/nginx_error.log error;
    #error_log  logs/error.log  notice;
    #error_log  logs/error.log  info;

    #pid        logs/nginx.pid;


    worker_rlimit_nofile 51200;

    events {
        use epoll;
        worker_connections  51200;
        multi_accept on;
    }

    http {
        include       mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx_access.log  main;

        server_names_hash_bucket_size 128;
        client_header_buffer_size 32k;
        large_client_header_buffers 4 32k;

        sendfile        on;
        tcp_nopush     on;
        tcp_nodelay    on;

        #keepalive_timeout  0;
        keepalive_timeout  65;

        #gzip  on;

        upstream registry {
            server 192.168.172.150:5000;
        }    

        ldap_server ldapserver {
            url ldap://xxxxxxxxx:389/OU=Users,DC=xxxxx,DC=it?samaccountname?sub?(objectClass=user);
            binddn xxxxx@xxxxxxxxxx.it;
            binddn_passwd xxxxxxxxxxxx
            group_attribute uniquemember;
            group_attribute_is_dn on;
        }

        server {
            listen       443;
            server_name  192.168.172.150;

            ssl          on;
            ssl_certificate /etc/nginx/ssl/nginx.crt;
            ssl_certificate_key /etc/nginx/ssl/nginx.key;

            client_max_body_size 0;

            chunked_transfer_encoding on;

            location /v2/ {
              auth_ldap "Forbidden";
              auth_ldap_servers ldapserver;
              add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

              proxy_pass                          http://registry;
              proxy_set_header  Host              \$http_host;   # required for docker client's sake
              proxy_set_header  X-Real-IP         \$remote_addr; # pass on real client's IP
              proxy_set_header  X-Forwarded-For   \$proxy_add_x_forwarded_for;
              proxy_set_header  X-Forwarded-Proto $scheme;
              proxy_read_timeout                  900;
            }

            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            }
        }
    
}
还有一个地方:如果之前有配置过V1的,可以保留,但最好分离出来,我是直接注掉了。
5.验证及启动启动nginx
/opt/nginx/sbin/nginx -t
    /opt/nginx/sbin/nginx -s /opt/nginx/conf/nginx.conf


registry源码编译及配置

1.checkout 源码:
git clone https://github.com/docker/distribution
    cd distribution
    git checkout v2.1.1
    godep restore ./...


2.安装registry,这里图省事,要编译重新设定下gopath即可
go get github.com/docker/distribution


3.配置registry,config-example.yml

version: 0.1
    log:
      fields:
        service: registry
    storage:
        cache:
            layerinfo: inmemory
        filesystem:
            rootdirectory: /home/jojo/registry
    http:
        addr: :5000
        secret: admin
    #    tls: 
    #      certificate: /etc/ssl/demoCA/certs/cacert.pem
    #      key: /etc/ssl/demoCA/private/cakey.pem
    #proxy:
    #  remoteurl: https://api.192.168.172.150.xip.io
    #  username: admin
    #  password: admin


这里注意我注掉的部分,因为前方已经有一层代理了,我们这里就没有必要设置tls了,否则,后端 会报tls: first record does not look like a TLS handshake
4.配置docker,增加一行
vi /etc/default/docker
    DOCKER_OPTS="--insecure-registry api.192.168.172.150.xip.io --tlsverify --tlscacert /etc/ssl/demoCA/certs/cacert.pem"

5.启动docker和registry
service docker start
    registry /home/jojo/register/config-example.yml


验证及排错

1.验证联通性:
curl -i -k -v https://admin:admin@api.192.168.172.150.xip.io/v2/

    * Hostname was NOT found in DNS cache
    *   Trying 192.168.172.150...
    * Connected to api.192.168.172.150.xip.io (192.168.172.150) port 443 (#0)
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * SSLv3, TLS handshake, Client hello (1):
    * SSLv3, TLS handshake, Server hello (2):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Server key exchange (12):
    * SSLv3, TLS handshake, Server finished (14):
    * SSLv3, TLS handshake, Client key exchange (16):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSL connection using ECDHE-RSA-AES256-GCM-SHA384
    * Server certificate:
    *        subject: C=CN; ST=beijing; O=self; OU=self; CN=*.192.168.172.150.xip.io; emailAddress=jack@126.com
    *        start date: 2015-09-18 13:54:11 GMT
    *        expire date: 2016-09-17 13:54:11 GMT
    *        issuer: C=CN; ST=beijing; L=beijing; O=self; OU=self; CN=*.192.168.172.150.xip.io; emailAddress=jack@126.com
    *        SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
    * Server auth using Basic with user 'admin'
    > GET /v2/ HTTP/1.1
    > Authorization: Basic YWRtaW46YWRtaW4=
    > User-Agent: curl/7.35.0
    > Host: api.192.168.172.150.xip.io
    > Accept: */*
    > 
    < HTTP/1.1 200 OK
    HTTP/1.1 200 OK
    * Server nginx/1.9.4 is not blacklisted
    < Server: nginx/1.9.4
    Server: nginx/1.9.4
    < Date: Fri, 18 Sep 2015 17:02:00 GMT
    Date: Fri, 18 Sep 2015 17:02:00 GMT
    < Content-Type: application/json; charset=utf-8
    Content-Type: application/json; charset=utf-8
    < Content-Length: 2
    Content-Length: 2
    < Connection: keep-alive
    Connection: keep-alive
    < Docker-Distribution-Api-Version: registry/2.0
    Docker-Distribution-Api-Version: registry/2.0
    < Docker-Distribution-Api-Version: registry/2.0
    Docker-Distribution-Api-Version: registry/2.0


2.验证docker login

docker login -u admin -p admin -e jackyuan@126 https://api.192.168.172.150.xip.io/v2/

    WARNING: login credentials saved in /root/.docker/config.json
    Login Succeeded


3.push镜像到私有registry
docker tag 91e54dfb1179 api.192.168.172.150.xip.io/ubuntu:trusty
    docker push api.192.168.172.150.xip.io/ubuntu:trusty

    The push refers to a repository [api.192.168.172.150.xip.io/ubuntu] (len: 1)
    91e54dfb1179: Image already exists 
    d74508fb6632: Image successfully pushed 
    c22013c84729: Image successfully pushed 
    d3a1f33e8a5a: Image successfully pushed 
    Digest: sha256:a731c12a4d21af384c4659666f177cd1e871646b95b9440d709ec4ee176145b2

4.查看是否上传
curl -i -k -v https://admin:admin@api.192.168.172.150.xip.io/v2/_catalog

    {"repositories":["ubuntu"]}

    cd /home/jojo/registry/docker/registry/v2 && ls
    blobs  repositories


这里就不往里面看了,深入registry后可以继续看里面的数据结构。

0 个评论

要回复文章请先登录注册