How and Where to Update Operative Config? and If I use older versions, I get the error still!

[ayush714]

I am getting this error, by using below code:-

ValueError: Configurable 'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.
  In file "gs://unifiedqa/models/large/operative_config.gin", line 83
    decoder/make_layer_stack.use_universal_transformer = False
    MODEL_SIZE = "large"
    BASE_PRETRAINED_DIR = "gs://unifiedqa/models/large"
    PRETRAINED_DIR = BASE_PRETRAINED_DIR
    MODEL_DIR = os.path.join(MODEL_DIR, MODEL_SIZE)

    model_parallelism, train_batch_size, keep_checkpoint_max = {
        "small": (1, 256, 16),
        "base": (2, 128, 8),
        "large": (8, 64, 4),
        "3B": (8, 16, 1),
        "11B": (8, 16, 1)}[MODEL_SIZE]
    tf.io.gfile.makedirs(MODEL_DIR)
    ON_CLOUD = False
    model = t5.models.MtfModel(
        model_dir=MODEL_DIR,
        tpu=None,
        model_parallelism=model_parallelism,
        batch_size=train_batch_size,
        sequence_length={"inputs": 128, "targets": 32},
        learning_rate_schedule=0.003,
        save_checkpoints_steps=5000,
        keep_checkpoint_max=keep_checkpoint_max if ON_CLOUD else None,
        iterations_per_loop=100,
    )
    FINETUNE_STEPS = 9

    logInfo("Started Training the model")
    start = time()
    model.finetune(
        mixture_or_task_name="qa_t5_meshs",
        pretrained_model_dir=PRETRAINED_DIR,
        finetune_steps=FINETUNE_STEPS
    )
    logInfo("Completed model training.", time_taken=time() - start)

How and Where to Update Operative Config? and If I use older versions, I get the error still!

[craffel]

The operative config is at gs://unifiedqa/models/large/operative_config.gin. It's in a GCS bucket. You don't have permission to update it; you need to get the UnifiedQA maintainers to update it. You could copy it locally and edit it yourself and refer to your local copy, though.

[ayush714]

and How to refer Only Operative config file and use models from bucket?

[craffel]

You can just remove the decoder/make_layer_stack.use_universal_transformer = False line. Just change the flag in your command to point to your local path to the operative config rather than gs://unifiedqa/models/large/operative_config.gin.

[ayush714]

Okay @craffel I am not using any command I am using manually fine tuning t5mesh, so where I can point it to my local operative config?

    MODEL_SIZE = "large"
    BASE_PRETRAINED_DIR = "gs://unifiedqa/models/large"
    PRETRAINED_DIR = BASE_PRETRAINED_DIR
    MODEL_DIR = os.path.join(MODEL_DIR, MODEL_SIZE)

    model_parallelism, train_batch_size, keep_checkpoint_max = {
        "small": (1, 256, 16),
        "base": (2, 128, 8),
        "large": (8, 64, 4),
        "3B": (8, 16, 1),
        "11B": (8, 16, 1)}[MODEL_SIZE]
    tf.io.gfile.makedirs(MODEL_DIR)
    ON_CLOUD = False
    model = t5.models.MtfModel(
        model_dir=MODEL_DIR,
        tpu=None,
        model_parallelism=model_parallelism,
        batch_size=train_batch_size,
        sequence_length={"inputs": 128, "targets": 32},
        learning_rate_schedule=0.003,
        save_checkpoints_steps=5000,
        keep_checkpoint_max=keep_checkpoint_max if ON_CLOUD else None,
        iterations_per_loop=100,
    )
    FINETUNE_STEPS = 9

    logInfo("Started Training the model")
    start = time()
    model.finetune(
        mixture_or_task_name="qa_t5_meshs",
        pretrained_model_dir=PRETRAINED_DIR,
        finetune_steps=FINETUNE_STEPS
    )
    logInfo("Completed model training.", time_taken=time() - start)

[craffel]

You need to copy the contents of BASE_PRETRAINED_DIR to a local path and set BASE_PRETRAINED_DIR to that path instead.

[ayush714]

Okay @craffel I done that but I am resolved with that error but the next error which I am getting which is

ValueError: exactly one of keep_prob and rate should be set
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/ml-experimentation-dev/research/qna_finetuning/code/run_t5mesh_train.py in <module>
     127 logInfo("Started Training the model")
     128 start = time()
---> 129 model.finetune(
     130     mixture_or_task_name="qa_t5_meshs",
     131     pretrained_model_dir=PRETRAINED_DIR,